Data Engineering Podcast

NOVEMBER 20, 2021

In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset. Can you start by giving an overview of the state of the market for data lakes today?

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JUNE 16, 2019

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. Interview Introduction How did you get involved in the area of data management?

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Data Engineering Podcast

MAY 11, 2020

Pulsar is a well engineered and robust platform for building the core of any system that relies on durable access to easily scalable streams of data. You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Can you start by giving an overview of what Pulsar is?

Cloud 100

Cloud Lambda Architecture Kafka Hadoop 100

Data Engineering Podcast

AUGUST 21, 2022

In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. Just connect it to your database/data warehouse/data lakehouse/whatever you’re using and let them do the rest.

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Data Engineering Podcast

JUNE 1, 2020

What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform. We talked last in November of 2018.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Monte Carlo

MARCH 14, 2023

Data ingestion is the process of collecting data from various sources and moving it to your data warehouse or lake for processing and analysis. It is the first step in modern data management workflows. Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley. There are trade-offs.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52