Data Engineering Podcast

JUNE 16, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Monte Carlo

OCTOBER 31, 2024

AI data engineers are data engineers that are responsible for developing and managing data pipelines that support AI and GenAI data products. Essential Skills for AI Data Engineers Expertise in Data Pipelines and ETL Processes A foundational skill for data engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

APRIL 7, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

JUNE 30, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

MAY 12, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

JUNE 9, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

FEBRUARY 11, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Government Machine Learning 147

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

JUNE 2, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

APRIL 14, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free!

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Monte Carlo

MARCH 19, 2025

Monte Carlo and Databricks double-down on their partnership, helping organizations build trusted AI applications by expanding visibility into the data pipelines that fuel the Databricks Data Intelligence Platform. Read on to discover how we’re helping organizations ensure reliability across the entire data + AI lifecycle.

Unstructured Data 52

Unstructured Data Data Pipeline High Quality Data Banking 52

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

MARCH 17, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 147

Data Lake High Quality Data Architecture Machine Learning 147

Data Engineering Podcast

JANUARY 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. webapps vs. data pipelines vs. exploratory analysis, etc.)

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

NOVEMBER 26, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Precisely

FEBRUARY 20, 2025

When your AI has access to all this high-quality data, you gain more relevant insights that help you power better decision-making and foster trust in AI outputs. This applies to both the development quality and performance characteristics of your data pipelines as well as the data quality and overlay governance for this process.

Data Integration 69

Data Integration Government Datasets Data Pipeline 69

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

DareData

JULY 29, 2020

With these points in mind, I argue that the biggest hurdle to the widespread adoption of these advanced techniques in the healthcare industry is not intrinsic to the industry itself, or in any way related to its practitioners or patients, but simply the current lack of high-quality data pipelines.

Data Pipeline 52

Data Pipeline Healthcare Medical Pipeline-centric 52

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Sponsored By: Starburst : ![Starburst

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

NOVEMBER 19, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Project 147

Project Data Lake SQL High Quality Data 147

Data Engineering Podcast

NOVEMBER 5, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

PostgreSQL 147

PostgreSQL Data Lake SQL High Quality Data 147

Data Engineering Weekly

FEBRUARY 2, 2025

The article advocates for a "shift left" approach to data processing, improving data accessibility, quality, and efficiency for operational and analytical use cases. The CDC approach addresses challenges like time travel, data validation, performance, and cost by replicating operational data to an AWS S3-based Iceberg Data Lake.

Data Engineering 58

Data Engineering Data Engineer Engineering Data Lake 58

DataKitchen

FEBRUARY 17, 2025

Current open-source frameworks like YAML-based Soda Core, Python-based Great Expectations, and dbt SQL are frameworks to help speed up the creation of data quality tests. They are all in the realm of software, domain-specific language to help you write data quality tests.

SQL 74

SQL Python Government Data Engineering 74

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

DECEMBER 17, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Ascend.io

APRIL 27, 2023

Going into the Data Pipeline Automation Summit 2023, we were thrilled to connect with our customers and partners and share the innovations we’ve been working on at Ascend. The summit explored the future of data pipeline automation and the endless possibilities it presents.

Data Pipeline 52

Data Pipeline Pipeline-centric Data Validation Data Engineering 52

Data Engineering Podcast

DECEMBER 10, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 130

Data Lake SQL High Quality Data Architecture 130

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

Monte Carlo

APRIL 24, 2023

Build vs buy orchestration tooling Unlike the other components we’ve discussed in Part 3, data pipelines don’t require orchestration to be considered functional—at least not at a foundational level. And data orchestration tools are generally easy to stand-up for initial use-cases. Missed Nishith’s 5 considerations?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. It is a good reminder to the data industry that we need to solve the fundamentals of data engineering to utilize AI better.

Data Engineering 103

Data Engineering Data Engineer Engineering Unstructured Data 103