Monte Carlo

NOVEMBER 21, 2024

Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way. That’s where data pipeline design patterns come in. Lambda Architecture Pattern 4.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Data Engineering Weekly

FEBRUARY 18, 2025

It addresses many of Kafka's challenges in analytical infrastructure. The combination of Kafka and Flink is not a perfect fit for real-time analytics; the integration of Kafka and Lakehouse is very shallow. How do you compare Fluss with Apache Kafka? Fluss and Kafka differ fundamentally in design principles.

Kafka 75

Kafka Lambda Architecture SQL Architecture 75

Rockset

FEBRUARY 6, 2019

That meant a system that was sufficiently nimble and powerful to execute fast SQL queries on raw data, essentially performing any needed transformations as part of the query step, and not as part of a complex data pipeline. In most cases, this would not be a single Spark job but a pipeline of Spark jobs.

Lambda Architecture 40

Lambda Architecture Architecture MongoDB Kafka 40

Data Engineering Podcast

AUGUST 21, 2022

Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Data stacks are becoming more and more complex.

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Monte Carlo

MARCH 14, 2023

For a use case like this, real-time data isn’t necessary, but reliable, regularly recurring data access is. Some data teams will leverage micro-batch strategies for time sensitive use cases. These involve data pipelines that will ingest data every few hours or even minutes.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52

ProjectPro

AUGUST 24, 2021

Data Sourcing: Building pipelines to source data from different company data warehouses is fundamental to the responsibilities of a data engineer. So, work on projects that guide you on how to build end-to-end ETL/ELT data pipelines. Also, explore other alternatives like Apache Hadoop and Spark RDD.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52