Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. How have the requirements of generative AI shifted the demand for streaming data systems? Data lakes are notoriously complex.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Data lakes are notoriously complex.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. What are the experimental methods that you are using to gain understanding in the opportunities and practical limits of those systems? What do you have planned for the future of your academic research?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. We feel your pain. It ends up being anything but that.

Building 147

Building Machine Learning Metadata Data Integration 147

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineer 363

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.    He then worked at the casual games company Zynga, building their in-game advertising platform.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Engineering at Meta

DECEMBER 6, 2023

Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration. Neither WhatsApp nor Secret Conversations operated in this manner, and we didn’t want all users to have to rely on a device-side storage system.

Building 145

Building Designing Consulting Accessible 145

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Why Postgres? Rudderstack]([link] Introducing RudderStack Profiles.

Database 130

Database Building PostgreSQL BI 130

Confluent

FEBRUARY 6, 2024

Confluent enables real-time, reliable, scalable, and secure communication between IoT devices, applications, and backend systems. Streamline data processing and unlock analytics to boost productivity and time to market while lowering infrastructure costs.

Building 119

Building Data Data Process Systems 119

Engineering at Meta

DECEMBER 10, 2024

Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.

Systems 74

Systems Architecture Engineering Data Pipeline 74

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

The Pragmatic Engineer

AUGUST 22, 2023

Q: “I play video games, and I was wondering how building games compares to more ‘standard’ software development. At Skype, I worked on the team building Skype for the Xbox One console, and several engineers on that team had previously worked in games. To get the full issues, twice a week, subscribe here.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” And COBOL was just one of many attempts.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. What is driving the broad adoption of columnar stores as a separate environment from transactional systems? Datafold : ![Datafold]([link]

Database 189

Database BI SQL Machine Learning 189

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. Going from idea, to adding, to building and shipping this to all customers, is something you rarely see in bigger companies. We cover one out of four topics in today’s subscriber-only The Pulse issue.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Who needs to be involved in the process of defining and developing that program? Go to dataengineeringpodcast.com/dagster today to get started.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Rudderstack]([link] Introducing RudderStack Profiles.

BI 208

BI SQL Data Machine Learning 208

Data Engineering Podcast

MAY 21, 2023

Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

The Pragmatic Engineer

FEBRUARY 29, 2024

But first, a few current cases of systems whose developers didn’t: In Sweden, card payments are down at a leading supermarket chain. Airline Avianca printed tickets dated as 3/1 instead of 2/29, thanks to their system not accounting for the leap day. The system was almost fully restored before noon.”

Software Engineer 220

Software Engineer Software Engineering Banking Engineering 220

The Pragmatic Engineer

APRIL 6, 2023

This created an opportunity to build job sites which collect this data, make it easy to browse, and allow job seekers to apply to jobs paying at or above a certain level. For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing.

Software Engineer 194

Software Engineer Software Engineering Datasets Database 194

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. Replay traffic enabled us to test our new systems and algorithms at scale before launch, while also making the traffic as realistic as possible.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Snowflake

APRIL 8, 2024

In 2020, Snowflake announced a new global competition to recognize the work of early-stage startups building their apps — and their businesses — on Snowflake, offering up to $250,000 in investment as the top prize. Let’s meet the 10 companies competing for the 2024 Snowflake Startup Challenge crown! SignalFlare.ai

Pipeline-centric 129

Pipeline-centric Food Healthcare Unstructured Data 129

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Your first 30 days are free!

Database 162

Database Technology Data Lake High Quality Data 162

The Pragmatic Engineer

APRIL 19, 2023

I still remember being in a meeting where a Very Respected Engineer was explaining how they are building a project, and they said something along the lines of "and, of course, idempotency is non-negotiable." I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager.

Software Engineer 295

Software Engineer Software Engineering Engineering Media 295

Data Engineering Podcast

SEPTEMBER 3, 2023

The dlt project was created to eliminate overhead and bring data integration into your full control as a library component of your overall data system. In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today.

Data Integration 162

Data Integration BI SQL Python 162

Monte Carlo

NOVEMBER 22, 2024

While your current setup might feel like it’s held together with duct tape and hope, there are ways to build up data engineering pipelines—automated workflows that collect, clean, and deliver data to the right destinations—without burning everything down and starting over. It means you’re scaling!

Data Engineering 52

Data Engineering Data Engineer Building Engineering 52

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This technique facilitates validation on multiple fronts.

Utilities 139

Utilities Systems Architecture Coding 139