Data Engineering Podcast

JUNE 1, 2020

Summary Data lakes offer a great deal of flexibility and the potential for reduced cost for your analytics, but they also introduce a great deal of complexity. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JUNE 16, 2019

Summary Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Monte Carlo

NOVEMBER 21, 2024

In this guide, we’ll explore the patterns that can help you design data pipelines that actually work. Table of Contents Common Data Pipeline Design Patterns Explained 1. Lambda Architecture Pattern 4. Kappa Architecture Pattern 5. Data Mesh Pattern 8. The data lakehouse has got you covered! ​​

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Data Engineering Weekly

FEBRUARY 18, 2025

Fluss is a compelling new project in the realm of real-time data processing. I spoke with Jark Wu , who leads the Fluss and Flink SQL team at Alibaba Cloud, to understand its origins and potential. Among the 20,000 Flink SQL jobs at Alibaba, only 49% of columns of Kafka data are read on average.

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Rockset

FEBRUARY 6, 2019

That meant a system that was sufficiently nimble and powerful to execute fast SQL queries on raw data, essentially performing any needed transformations as part of the query step, and not as part of a complex data pipeline. Most processing in the Lambda architecture happens in the pipeline and not at query time.

Lambda Architecture 40

Lambda Architecture Architecture MongoDB Kafka 40

ProjectPro

AUGUST 24, 2021

This data engineering project uses the following big data stack - Azure Structured Query Language (SQL) Database instance for persistent storage; to store forecasts and historical distribution data. Google BigQuery receives the structured data from workers. Upload it to Azure Data lake storage manually.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52