Data Engineering Weekly

FEBRUARY 18, 2025

It addresses many of Kafka's challenges in analytical infrastructure. The combination of Kafka and Flink is not a perfect fit for real-time analytics; the integration of Kafka and Lakehouse is very shallow. How do you compare Fluss with Apache Kafka? Fluss and Kafka differ fundamentally in design principles.

Kafka 75

Kafka Lambda Architecture SQL Architecture 75

Data Engineering Podcast

MAY 11, 2020

Pulsar is a well engineered and robust platform for building the core of any system that relies on durable access to easily scalable streams of data. What is Pulsar’s role in the lifecycle of data and where does it fit in the overall ecosystem of data tools? Why is streaming data such an important capability?

Cloud 100

Cloud Lambda Architecture Kafka Hadoop 100

Data Engineering Podcast

DECEMBER 31, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. How do you represent a stream on-disk?

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Data Engineering Podcast

AUGUST 21, 2022

In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. Can you describe what is driving the adoption of real-time analytics?

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Data Engineering Podcast

JUNE 1, 2020

What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform. We talked last in November of 2018.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Monte Carlo

MARCH 14, 2023

Data ingestion is the process of collecting data from various sources and moving it to your data warehouse or lake for processing and analysis. It is the first step in modern data management workflows. Some data teams will leverage micro-batch strategies for time sensitive use cases.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52