Monte Carlo

NOVEMBER 21, 2024

Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way. That’s where data pipeline design patterns come in. Lambda Architecture Pattern 4.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Data Engineering Podcast

NOVEMBER 20, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Can you start by giving an overview of the state of the market for data lakes today?

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

AUGUST 21, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Can you describe what is driving the adoption of real-time analytics? When is Rockset the wrong choice?

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Data Engineering Podcast

JUNE 16, 2019

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Data Engineering Podcast

MAY 11, 2020

You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Tidy Data is the DataOps monitoring platform that you’ve been missing. You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data?

Cloud 100

Cloud Lambda Architecture Kafka Hadoop 100

Data Engineering Podcast

JUNE 1, 2020

How does the introduction of a universal SQL layer change the staffing requirements for building and maintaining a data lake? What are the advantages of a data lake over a data warehouse if everything is being managed via SQL anyway?

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Cloudera

NOVEMBER 2, 2020

Users today are asking ever more from their data warehouse. As an example of this, in this post we look at Real Time Data Warehousing (RTDW), which is a category of use cases customers are building on Cloudera and which is becoming more and more common amongst our customers. What is Real Time Data Warehousing?

Data Warehouse 97

Data Warehouse Kafka Lambda Architecture Telecommunication 97