Data Engineering Weekly

FEBRUARY 18, 2025

I spoke with Jark Wu , who leads the Fluss and Flink SQL team at Alibaba Cloud, to understand its origins and potential. Jark is a key figure in the Apache Flink community, known for his work in building Flink SQL from the ground up and creating Flink CDC and Fluss. It addresses many of Kafka's challenges in analytical infrastructure.

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Monte Carlo

NOVEMBER 21, 2024

Lambda Architecture Pattern 4. Kappa Architecture Pattern 5. Lambda Architecture Pattern Here’s where things get interesting. Lambda architecture is like having both a regular washing machine for your weekly loads AND that magical instant-wash machine. Batch Processing Pattern 2.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Rockset

FEBRUARY 6, 2019

That meant a system that was sufficiently nimble and powerful to execute fast SQL queries on raw data, essentially performing any needed transformations as part of the query step, and not as part of a complex data pipeline. A common implementation would have large batch jobs in Hadoop complemented by an update stream stored in Apache Kafka.

Lambda Architecture 40

Lambda Architecture Architecture MongoDB Kafka 40

Data Engineering Podcast

JUNE 1, 2020

In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform. How does the introduction of a universal SQL layer change the staffing requirements for building and maintaining a data lake? How is the SQL layer in Upsolver implemented?

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

AUGUST 21, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Cloudera

NOVEMBER 2, 2020

Data streamed in is queryable in conjunction with historical data, avoiding need for Lambda Architecture. Figure 1 below shows a standard architecture for a Real-Time Data Warehouse. SQL editor for running Hive and Impala queries. SQL editor for running Impala+Kudu queries. with low latency and high concurrency.

Data Warehouse 97

Data Warehouse Kafka Lambda Architecture Telecommunication 97

Data Engineering Weekly

JULY 9, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Each architectural pattern has its limitation. Write SQL queries without learning SQL?

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52