Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JUNE 1, 2020

Summary Data lakes offer a great deal of flexibility and the potential for reduced cost for your analytics, but they also introduce a great deal of complexity. In order to bring the DBA into the new era of data management the team at Upsolver added a SQL interface to their data lake platform.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

JUNE 16, 2019

Summary Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Monte Carlo

NOVEMBER 21, 2024

In this guide, we’ll explore the patterns that can help you design data pipelines that actually work. Table of Contents Common Data Pipeline Design Patterns Explained 1. Lambda Architecture Pattern 4. Kappa Architecture Pattern 5. Data Mesh Pattern 8. Batch Processing Pattern 2.

52

52

Monte Carlo

MARCH 14, 2023

Data ingestion is the process of collecting data from various sources and moving it to your data warehouse or lake for processing and analysis. It is the first step in modern data management workflows. Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52

ProjectPro

AUGUST 24, 2021

Data Warehousing: Data warehousing utilizes and builds a warehouse for storing data. A data engineer interacts with this warehouse almost on an everyday basis. Data Analytics: A data engineer works with different teams who will leverage that data for business solutions.

Data Engineer 52

Data Engineer Data Engineering Coding Project 52