Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. How has that changed the architectural approach to CDPs? Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Weekly

FEBRUARY 18, 2025

Kafka is designed for streaming events, but Fluss is designed for streaming analytics. Architecture Difference The first difference is the Data Model. The fourth difference is the Lakehouse Architecture. Fluss embraces the Lakehouse Architecture. It excels in event-driven architectures and data pipelines.

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Towards Data Science

JANUARY 16, 2024

My personal take on justifying the existence of Data Mesh A senior stakeholder at one my projects mentioned that they wanted to decentralise their data platform architecture and democratise data across the organisation. When I heard the words ‘decentralised data architecture’, I was left utterly confused at first!

Data Warehouse 95

Data Warehouse Data Architecture Relational Database NoSQL 95

DataKitchen

AUGUST 19, 2021

Below is our fourth post (4 of 5) on combining data mesh with DataOps to foster innovation while addressing the challenges of a decentralized architecture. We’ve covered the basic ideas behind data mesh and some of the difficulties that must be managed. Figure 1: Data requirements for phases of the drug product lifecycle.

Pharmaceutical 98

Pharmaceutical Data Lake Data Warehouse Raw Data 98

Striim

AUGUST 22, 2024

“Sometimes there’s so much data that old batch processing (late at night once a day or once a week) just doesn’t have time to move all data and hence the only way to do it is trickle feed data via CDC,” says Dmitriy Rudakov, Director of Solution Architecture at Striim.

Data Warehouse 52

Data Warehouse Retail Machine Learning Database 52

Edureka

FEBRUARY 7, 2023

It allows developers to query external data from supported data stores transparently, regardless of the storage architecture of the external data store. One can use polybase: From Azure SQL Database or Azure Synapse Analytics, query data kept in Hadoop, Azure Blob Storage, or Azure Data Lake Store.

Data Engineering 52

Data Engineering Data Engineer Engineering Hadoop 52

Rockset

MARCH 19, 2020

With all of these stream processing and real-time data store options, though, also comes questions for when each should be used and what their pros and cons are. I hope by the end you find yourself better informed and less confused about the real-time analytics landscape and are ready to dive in to it for yourself.

Kafka 40

Kafka Database Process SQL 40