Data Engineering Podcast

MAY 1, 2022

He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the maintenance burden of data integration workflows. Struggling with broken pipelines? What do you have planned for the future of Matillion?

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

APRIL 19, 2021

Summary Most of the time when you think about a data pipeline or ETL job what comes to mind is a purely mechanistic progression of functions that move data from point A to point B. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

Data Engineering Podcast

JUNE 1, 2021

Summary Every part of the business relies on data, yet only a small team has the context and expertise to build and maintain workflows and data pipelines to transform, clean, and integrate it. RudderStack’s smart customer data pipeline is warehouse-first. Closing Announcements Thank you for listening!

Data Pipeline 100

Data Pipeline Data Warehouse Data Engineer Data 100

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. They provide an AWS-native, serverless, data infrastructure that installs in your VPC. Datacoral helps data engineers build and manage the flow of data pipelines without having to manage any infrastructure.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Data Engineering Podcast

JANUARY 18, 2021

Summary The data warehouse has become the central component of the modern data stack. This is an interesting conversation about the importance of the data warehouse and how it can be used beyond just internal analytics. How do you keep data up to date between the warehouse and downstream systems?

Data Warehouse 100

Data Warehouse BI Data Data Engineer 100

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). To start, can you share your definition of what constitutes a "Data Lakehouse"? Closing Announcements Thank you for listening!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. There are times when the data is structured , but it is often messy since it is ingested directly from the data source. What is Data Warehouse? . Data Warehouse in DBMS: .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Ascend.io

APRIL 28, 2023

Data pipelines are having a moment — at least, that is, within the data world. That’s because as more and more businesses are adopting a data-driven mindset, the movement of data into and within organizations has never been a bigger priority.

Data Pipeline 52

Data Pipeline Data Data Warehouse Data Engineer 52

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Christophe Blefari

MARCH 15, 2024

Understand how BigQuery inserts, deletes and updates — Once again Vu took time to deep dive into BigQuery internal, this time to explain how data management is done. Pandera, a data validation library for dataframes, now supports Polars. It's inspirational.

Metadata 272

Metadata Datasets Data Software Engineering 272

Data Engineering Podcast

DECEMBER 16, 2018

As the organization grows and gains more customers, the requirements for that pipeline will change. In this episode Christian Heinzmann, Head of Data Warehousing at Grubhub, discusses the various requirements for data pipelines and how the overall system architecture evolves as more data is being processed.

Data Pipeline 100

Data Pipeline Data Lake Data Warehouse Designing 100

Data Engineering Podcast

NOVEMBER 16, 2020

In this episode CEO and co-founder of Isima Darshan Rawal explains how the biOS platform is architected to enable ease of use, the challenges that were involved in building an entirely new system from scratch, and how it can integrate with the rest of your data platform to allow for incremental adoption. What is the story behind the name?

Data Management 100

Data Management Management BI Business Intelligence 100

Data Engineering Podcast

APRIL 9, 2023

When it was difficult to wire together the event collection, data modeling, reporting, and activation it made sense to buy monolithic products that handled every stage of the customer data lifecycle. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Engineer 225

Data Engineer Data Engineering Python Engineering 225

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP).

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

JUNE 4, 2023

In this episode Tevje Olin explains how the platform is implemented, the features that it provides to reduce the amount of effort required to keep your pipelines running, and how you can start using it in your own team. Can you describe what Agile Data Engine is and the story behind it? What does CI/CD look like for a data warehouse?

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Data Engineering Podcast

JUNE 9, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data observability has been gaining adoption for a number of years now, with a large focus on data warehouses.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

DECEMBER 28, 2022

In this episode he shares his thoughts on the strategic and tactical elements of moving your work as a data professional from being task-oriented to being product-oriented and the long term improvements in your productivity that it provides. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. ETL has typically been carried out utilizing data warehouses and on-premise ETL tools.

AWS 52

AWS Data Management ETL Tools Management 52

Data Engineering Podcast

DECEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Pricing for SQLake is simple.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

MAY 14, 2023

In this episode the host Tobias Macey shares his reflections on recent experiences where the abstractions leaked and some observances on how to deal with that situation in a data platform architecture. What do you have planned for the future of your data platform? What do you have planned for the future of your data platform?

Data Lake 147

Data Lake Machine Learning Data Warehouse AWS 147

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Data Engineering Podcast

MAY 28, 2023

Ghalib Suleiman has been on both sides of this equation and joins the show to share his hard-won wisdom about how to start and grow a data team in the early days of company growth. Can you start by sharing your conception of the responsibilities of a data team? When is it more practical to outsource the data work?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Weekly

OCTOBER 11, 2024

The data world is abuzz with speculation about the future of data engineering and the successor to the celebrated modern data stack. While the modern data stack has undeniably revolutionized data management with its cloud-native approach, its complexities and limitations are becoming increasingly apparent.

Manufacturing 109

Manufacturing Transportation Data Warehouse Unstructured Data 109

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Can you describe what your conception of a data contract is? Closing Announcements Thank you for listening!

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Ascend.io

APRIL 12, 2023

From this research, we developed a framework with a sequence of stages to implement data integrity quickly and measurably via data pipelines. Table of Contents Why does data integrity matter? At every level of a business, individuals must trust the data, so they can confidently make timely decisions. Let’s explore!

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

Data Engineering Podcast

JANUARY 15, 2023

In this episode Srujan Akula explains how the system is implemented and how you can start using it today with your existing data systems. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Your flagship (only?)

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

JANUARY 8, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data and analytics leaders, 2023 is your year to sharpen your leadership skills, refine your strategies and lead with purpose. Monte Carlo]([link] Struggling with broken pipelines? Missing data?

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Monte Carlo

JULY 9, 2024

As more organizations race to adopt GenAI and build AI-powered data products, DuckDB provides emerging applications as the storage layer of RAG knowledge bases to streamline and expedite data management. And it’s absolutely vital for MotherDuck users to know when data quality issues occur.

Data Warehouse 69

Data Warehouse Data Pipeline SQL Database 69

Data Engineering Podcast

DECEMBER 29, 2022

In this episode she shares the strategic and tactical elements of how to make more effective use of the technical and organizational resources that are available to you for getting work done with data. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform!

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

MAY 21, 2023

In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162