Uber Engineering
JUNE 22, 2021
Introduction. Uber relies on a containerized microservice architecture. Our need for computational resources has grown significantly over the years, as a consequence of business’ growth. It is an important goal now to increase the efficiency of our computing resources. Broadly … The post Efficient and Reliable Compute Cluster Management at Scale appeared first on Uber Engineering Blog.
Start Data Engineering
JUNE 25, 2021
Introduction Objective Setup Pre-requisites Project 1. ETL Code 2. Test 3. Scheduler 4. Presentation 4.1. Formatting, Linting, and Type checks 4.2. Architecture Diagram 4.3. README.md 5. Adding Dashboard to your Profile Future Work Tear down infra Conclusion Further Reading References Introduction Building a data project for your portfolio is hard. Getting hiring managers to read through your Github code is even harder.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confluent
JUNE 23, 2021
Al data til folket (all data to the people) is a compelling proposition in an enterprise context. Yet the ability to quickly address integration challenges and deliver data to those […].
Data Engineering Podcast
JUNE 25, 2021
Summary Data Engineering is a broad and constantly evolving topic, which makes it difficult to teach in a concise and effective manner. Despite that, Daniel Molnar and Peter Fabian started the Pipeline Academy to do exactly that. In this episode they reflect on the lessons that they learned while teaching the first cohort of their bootcamp how to be effective data engineers.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
DataKitchen
JUNE 24, 2021
In May 2021 at the CDO & Data Leaders Global Summit, DataKitchen sat down with the following data leaders to learn how to use DataOps to drive agility and business value. Kurt Zimmer, Head of Data Engineering for Data Enablement at AstraZeneca. Ryan Chapin, Former Executive Manager, Advanced Additive Design, Chief Product and Portfolio Manager, GE Aviation.
Cloudera
JUNE 25, 2021
Introduction. Many Cloudera customers are making the transition from being completely on-prem to cloud by either backing up their data in the cloud, or running multi-functional analytics on CDP Public cloud in AWS or Azure. The Replication Manager service facilitates both disaster recovery and data migration across different environments. Using easy-to-define policies, Replication Manager solves one of the biggest barriers for the customers in their cloud adoption journey by allowing them to mov
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Data Engineering Podcast
JUNE 22, 2021
Summary The database is the core of any system because it holds the data that drives your entire experience. We spend countless hours designing the data model, updating engine versions, and tuning performance. But how confident are you that you have configured it to be as performant as possible, given the dozens of parameters and how they interact with each other?
ProjectPro
JUNE 22, 2021
This blog will help you master the fundamentals of classification machine learning algorithms with their pros and cons. You will also explore some exciting machine learning project ideas that implement different types of classification algorithms. So, without much ado, let's dive in. Imagine that the pandemic is over and today is a weekday. All the schools, colleges, and offices are open, and you should reach your institution by 8 A.M.
Netflix Tech
JUNE 22, 2021
A Script Authoring Specification By: Bhanu Srikanth, Andy Swan, Casey Wilms, Patrick Pearson The Art of Dubbing and Subtitling Dubbing and subtitling are inherently creative processes. At Netflix, we strive to make shows as joyful to watch in every language as in the original language, whether a member watches with original or dubbed audio, closed captions, forced narratives, subtitles or any combination they prefer.
Confluent
JUNE 25, 2021
I’m excited to announce that Confluent was presented with the Databricks ISV Momentum Award at the Databricks Partner Executive Summit last month. This award is given to the partner whose […].
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
DataKitchen
JUNE 25, 2021
In this five-module course, Mike Lampa & Chris Bergh teach data professionals to plan their organization's DataOps program for low errors & fast deployment. The post Standing Up a DataOps Program for Practitioners first appeared on DataKitchen.
Teradata
JUNE 20, 2021
Open Banking is re-shaping the landscape of financial services and introducing new types of risks extending beyond data security. Secure open banking is everyone’s responsibility.
Netflix Tech
JUNE 25, 2021
By Gim Mahasintunan on behalf of Data Platform Engineering. Supporting a rapidly growing base of engineers of varied backgrounds using different data stores can be challenging in any organization. Netflix’s internal teams strive to provide leverage by investing in easy-to-use tooling that streamlines the user experience and incorporates best practices.
Confluent
JUNE 22, 2021
To the developer or architect seeking to provide their business with as much value as possible, what is the best way to start working with data in motion? Choosing Apache […].
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Cloudera
JUNE 24, 2021
CDP Operational Database Experience (COD) is a PaaS offering on the Cloudera Data Platform (CDP). COD enables you to create a new operational database with a few clicks and auto-scales based on your workload. Behind the scenes, COD automatically manages cluster deployment and configuration, reducing overheads related to setting up new database instances.
Monte Carlo
JUNE 24, 2021
Monte Carlo , the data reliability company, today announced two new strategic hires to its leadership team: Daniel Day , Head of Marketing, and Jordan Van Horn , Head of Revenue. With experience leading award-winning go-to-market teams at Snowflake and Segment, Day and Van Horn share a deep expertise in the data industry and will help Monte Carlo meet the growing demands as the industry leader in Data Observability.
Teradata
JUNE 24, 2021
The connected cloud data warehouse is fundamental to Data Mesh implementation in large and complex organisations. Find out why.
Rock the JVM
JUNE 23, 2021
Explore ZIO's unique fiber model for concurrency: see how it stands out from other effect libraries in the Scala ecosystem
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
FreshBI
JUNE 23, 2021
It’s a jungle out there Back in the day- when I was stuck on a DAX problem, I used to toggle through the IntelliSense in PowerBI one letter at a time. I’ve learned much since then and in this blog I’d like to share my experience with using NORM.DIST in Dax. A: ABS ACOS ACOSH … B: BETA.DIST BETA.INV BLANK Etc…. Hours wasted. Mistakes were made A MUCH better use of my time would have been reviewing quality solutions to real world problems.
Zalando Engineering
JUNE 23, 2021
Introduction In our previous post about Technology Choices at Zalando we spoke about a few problems with scaling technology selection in Tech companies. Since then, we have focused on the remaining categories of the Tech Radar beyond languages and the Tech Radar contribution process. Now, we'd like to reflect on our lessons learned, which you can use when designing technology selection processes.
DataKitchen
JUNE 23, 2021
Cognizant's JP Thakur & DataKitchen's Chris Bergh discuss how DataOps sets the foundation for Data Modernization initiatives enabling continuous data & insight. The post The Role of DataOps in Data Modernization first appeared on DataKitchen.
Rock the JVM
JUNE 23, 2021
Explore ZIO's unique fiber model for concurrency: see how it stands out from other effect libraries in the Scala ecosystem
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Teradata
JUNE 22, 2021
When undertaking the expense & time to execute a proof of concept on your journey to the cloud, make sure the efforts are well defined & drive an actionable outcome.
ProjectPro
JUNE 19, 2021
If you have heard of Machine Learning and Deep Learning, you must have also heard about cost (error or loss) functions. But, even if you haven't, fret not! The cost function, in simple words, is a way of measuring the performance of a Machine Learning model by attributing a cost to every 'mistake' or wrong prediction that the model makes. But as we know from personal experience, one can gain little by simply knowing that a mistake has occurred or how many, for that matter, if you have no clue ho
Preset
JUNE 25, 2021
Analyzing data that’s frequently updated? A comprehensive walkthrough of how to analyze and visualize ClickHouse data in Superset.
RudderStack
JUNE 24, 2021
This blog talks in detail about RudderStack's product vision- what you can expect from RudderStack in the coming quarters in terms of features and experience.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Cloudera
JUNE 23, 2021
Climate change is a challenge for insurers in some obvious ways, such as stronger and more frequent natural disasters. Yet there are also more subtle risks to monitor, including changes to insured assets, risks, and exposures. Climate impacts the production quality and quantity of insured consumable goods, their location, and their supply chains. Climate change can also impact the insurance carrier as an enterprise itself—similarly to cyber risks, insurers underwrite cyber risks for their custom
ProjectPro
JUNE 24, 2021
According to Gartner , organizations can suffer a financial loss of up to 15 million dollars for the poor quality of data. As per McKinsey , 47% of organizations believe that data analytics has impacted the market in their respective industries. According to Forbes , in 2012 only 12% of Fortune 1000 companies reported having a CDO (Chief Data Officer).
RudderStack
JUNE 22, 2021
RudderStack product news update includes the latest security features- Mobile SDK Distribution, New Spreadsheet, Database Destinations, and RudderStack video content.
Expert insights. Personalized for you.
139 
Let's personalize your content