Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JUNE 1, 2020

Summary Data lakes offer a great deal of flexibility and the potential for reduced cost for your analytics, but they also introduce a great deal of complexity. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

JUNE 16, 2019

Summary Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Monte Carlo

NOVEMBER 21, 2024

In this guide, we’ll explore the patterns that can help you design data pipelines that actually work. Table of Contents Common Data Pipeline Design Patterns Explained 1. Lambda Architecture Pattern 4. Kappa Architecture Pattern 5. Data Mesh Pattern 8. Batch Processing Pattern 2.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Data Engineering Weekly

FEBRUARY 18, 2025

The fourth difference is the Lakehouse Architecture. Fluss embraces the Lakehouse Architecture. Fluss uses Lakehouse as a tiered storage, and data will be converted and tiered into data lakes periodically; Fluss only retains a small portion of recent data. What is the future roadmap for Fluss?

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Rockset

FEBRUARY 6, 2019

Traditional Data Processing: Batch and Streaming MapReduce, most commonly associated with Apache Hadoop, is a pure batch system that often introduces significant time lag in massaging new data into processed results. Most processing in the Lambda architecture happens in the pipeline and not at query time.

Lambda Architecture 40

Lambda Architecture Architecture MongoDB Kafka 40

Monte Carlo

MARCH 14, 2023

Data from these sources are often ingested into a cloud-based data warehouse or data lake , where they can then be mined for information and insights. Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley. Some data teams will leverage micro-batch strategies for time sensitive use cases.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52