Data Engineering Podcast

NOVEMBER 20, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. The Lambda architecture has largely been abandoned, so what is the answer for today’s data lakes?

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

AUGUST 21, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Data Engineering Podcast

MAY 11, 2020

You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Tidy Data is the DataOps monitoring platform that you’ve been missing. You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data?

Cloud 100

Cloud Lambda Architecture Kafka Hadoop 100

Data Engineering Podcast

JUNE 16, 2019

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. What are some of the common antipatterns in data lake implementations and how does Delta Lake address them?

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Data Engineering Podcast

JUNE 1, 2020

How does the introduction of a universal SQL layer change the staffing requirements for building and maintaining a data lake? What are the advantages of a data lake over a data warehouse if everything is being managed via SQL anyway?

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Cloudera

NOVEMBER 2, 2020

Users today are asking ever more from their data warehouse. As an example of this, in this post we look at Real Time Data Warehousing (RTDW), which is a category of use cases customers are building on Cloudera and which is becoming more and more common amongst our customers. What is Real Time Data Warehousing?

Data Warehouse 96

Data Warehouse Kafka Lambda Architecture BI 96

Pinterest Engineering

MAY 2, 2023

This platform is also a key component for PinnerFormer work providing real-time user sequence data. Real-Time Indexing Pipeline The main goal of the real-time indexing pipeline is to enrich, store, and serve the last few relevant user actions as they come in. To explore life at Pinterest, visit our Careers page.

Lambda Architecture 80

Lambda Architecture Datasets Software Engineering Software Engineer 80