Data Engineering Podcast

JUNE 1, 2020

Summary Data lakes offer a great deal of flexibility and the potential for reduced cost for your analytics, but they also introduce a great deal of complexity. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Monte Carlo

NOVEMBER 21, 2024

In this guide, we’ll explore the patterns that can help you design data pipelines that actually work. Table of Contents Common Data Pipeline Design Patterns Explained 1. Lambda Architecture Pattern 4. Kappa Architecture Pattern 5. Data Mesh Pattern 8. Batch Processing Pattern 2.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Rockset

FEBRUARY 6, 2019

Traditional Data Processing: Batch and Streaming MapReduce, most commonly associated with Apache Hadoop, is a pure batch system that often introduces significant time lag in massaging new data into processed results. A common implementation would have large batch jobs in Hadoop complemented by an update stream stored in Apache Kafka.

Lambda Architecture 40

Lambda Architecture Architecture MongoDB Kafka 40

Monte Carlo

MARCH 14, 2023

Data from these sources are often ingested into a cloud-based data warehouse or data lake , where they can then be mined for information and insights. Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley. Some data teams will leverage micro-batch strategies for time sensitive use cases.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52

ProjectPro

AUGUST 24, 2021

This architecture shows that simulated sensor data is ingested from MQTT to Kafka. The data in Kafka is analyzed with Spark Streaming API, and the data is stored in a column store called HBase. Finally, the data is published and visualized on a Java-based custom Dashboard. This is called Hot Path.

Data Engineer 52

Data Engineer Data Engineering Coding Project 52