Data Engineering Podcast

APRIL 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. Data lakes are notoriously complex.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JUNE 30, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

JUNE 2, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. number/scale of systems, types of data, AI) Data governance can often become an exercise in boiling the ocean. Email hosts@dataengineeringpodcast.com ) with your story.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

APRIL 21, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

JANUARY 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data projects are notoriously complex. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. As you strive to push the limits of technical capacity in data systems, how does that impact the usability of the resulting systems?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

MAY 12, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

JUNE 9, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

FEBRUARY 4, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the core design philosophies that you rely on to prioritize the ongoing development of the project? Email hosts@dataengineeringpodcast.com ) with your story. Starburst : ![Starburst

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 3, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues for every part of your data workflow, from migration to deployment. Datafold has recently launched a 3-in-1 product experience to support accelerated data migrations.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

DECEMBER 24, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

APRIL 28, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MAY 26, 2024

When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. Can you start by sharing some of your experiences with data migration projects?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

MAY 5, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

MARCH 10, 2024

The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond that simple utility. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Data Engineering Podcast

FEBRUARY 11, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

MARCH 3, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

FEBRUARY 25, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

DECEMBER 10, 2023

Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project. Find simplicity in your most complex projects with Miro.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 26, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

DECEMBER 17, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 5, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. Email hosts@dataengineeringpodcast.com ) with your story. Sponsored By: Starburst : ![Starburst

Software Engineering 147

Software Engineering Software Engineer Engineering Data Lake 147

Ascend.io

JUNE 10, 2024

Snowflake’s partnership with Nvidia promises to revolutionize how we build and deploy AI-powered applications, equipping teams to deliver on AI projects faster than ever before. Snowflake’s investment in expanding data engineering capabilities is a game-changer.

Data Engineer 52

Data Engineer Data Engineering Engineering High Quality Data 52

Monte Carlo

MAY 25, 2023

Benefits of Data Orchestration Data orchestration can yield significant benefits for organizations—from driving efficiency across the data team to improving relationships across business domains. Automating data workflows. Data quality issues introduced by faster data workflows.

Data Pipeline 52

Data Pipeline Data Workflow Data Data Governance 52

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Monte Carlo

JULY 27, 2023

These specialists are also commonly referred to as data reliability engineers. To be successful in their role, data quality engineers will need to gather data quality requirements (mentioned in 65% of job postings) from relevant stakeholders.

Engineering 52

Engineering Healthcare Data Warehouse Scala 52

DataKitchen

JANUARY 3, 2022

When the project scope and schedule are fixed, managers have to add more staff to keep up with the workload. . These groups tend to be more insulated from their customer’s actual needs, so they take many months to deliver, and when a project does get done, it usually requires rework. Some IT teams are fantastic.

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98

Monte Carlo

APRIL 24, 2023

While this worked well for a time, the data infrastructure team quickly found itself needing to support hundreds of thousands of pipelines across the company. Airflow was an upcoming project at the time that supported most of the use-cases we needed.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52