Data Engineering Podcast

JUNE 9, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

JUNE 2, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst : ![Starburst

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

NOVEMBER 5, 2023

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

PostgreSQL 147

PostgreSQL Data Lake SQL High Quality Data 147

Striim

APRIL 18, 2025

Many organizations struggle with: Inconsistent data formats : Different systems store data in varied structures, requiring extensive preprocessing before analysis. Siloed storage : Critical business data is often locked away in disconnected databases, preventing a unified view. Start Your Free Trial | Schedule a Demo

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

MAY 12, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

NOVEMBER 19, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake SQL High Quality Data 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. You shouldn't have to throw away the database to build with fast-changing data.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. With Materialize, you can!

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

DECEMBER 3, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. SQL Server version upgrade) Section 2: Types of Migrations for Infrastructure Focus Storage migration: Moving data between systems (HDD to SSD, SAN to NAS, etc.) Starburst : ![Starburst

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

DECEMBER 17, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake SQL High Quality Data Architecture 130

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Monte Carlo

OCTOBER 31, 2024

The foundational skills are similar between traditional data engineers and AI data engineers are similar, with AI data engineers more heavily focused on machine learning data infrastructure, AI-specific tools, vector databases, and LLM pipelines. Let’s dive into the tools necessary to become an AI data engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period. Both integrations will be available early 2024.

Kafka 64

Kafka Database High Quality Data Cloud 64

DataKitchen

MARCH 20, 2025

Process-centric data teams focus their energies predominantly on orchestrating and automating workflows. They have demonstrated that robust, well-managed data processing pipelines inevitably yield reliable, high-quality data.

Pipeline-centric 65

Pipeline-centric Database-centric Process Data 65

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis. The team at Audio Analytic are working to impart a sense of hearing to our myriad devices with their sound recognition technology.

Data Collection 100

Data Collection Management High Quality Data Metadata 100

U-Next

FEBRUARY 27, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

Precisely

APRIL 7, 2025

Use case (Retail): As an example, imagine a retail company has a customer database with names and addresses, but many records are missing full address information. The solution: They use a data appending process to match their existing data with a third-party database that contains full street addresses. Plan for it.

Retail 74

Retail Datasets Data Portfolio 74

U-Next

MARCH 7, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Database Bytes Recruitment 98

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were. Those systems have been taught to normalize the data for storage on their own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Databand.ai

MAY 30, 2023

Data quality refers to the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context. High-quality data is essential for making well-informed decisions, performing accurate analyses, and developing effective strategies.

Data Cleanse 85

Data Cleanse Datasets Data Governance Data Validation 85

Precisely

JULY 12, 2024

However, simply having high-quality data does not, of itself, ensure that an organization will find it useful. Data observability: P revent business disruption and costly downstream data and analytics issues using intelligent technology that proactively alerts you to data anomalies and outliers.

Data Integration 105

Data Integration Datasets Data Data Governance 105

Towards Data Science

MARCH 29, 2023

Great reads on modeling, processes, and leadership Photo by Emil Widlund on Unsplash At the very start of my journey in data, I thought I was going to be a data scientist, and my first foray into data was centered on studying statistics and linear algebra, not software engineering or database management.

Data Warehouse 96

Data Warehouse BI Database Healthcare 96

Data Engineering Weekly

AUGUST 25, 2024

[link] Murat: Understanding the Performance Implications of Storage-Disaggregated Databases The separation of storage and computing certainly brings a lot of flexibility in operating data stores. The author writes an overview of the performance implication of disaggregated systems compared to traditional monolithic databases.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Precisely

SEPTEMBER 25, 2023

Data validation performs a check against existing values in a database to ensure that they fall within valid parameters. Data enrichment is the process of enhancing your data by appending relevant context from additional sources – improving its overall value, accuracy, and usability.

Data Validation 99

Data Validation Process Raw Data Data Cleanse 99

Monte Carlo

FEBRUARY 27, 2024

Organizations need to connect LLMs with their proprietary data and business context to actually create value for their customers and employees. They need robust data pipelines, high-quality data, well-guarded privacy, and cost-effective scalability. Data engineers. Who can deliver?

Data Engineering 59

Data Engineering Data Engineer Engineering High Quality Data 59

DataKitchen

NOVEMBER 4, 2021

It’s too hard to change our IT data product. Can we create high-quality data in an “answer-ready” format that can address many scenarios, all with minimal keyboarding? . “I I get cut off at the knees from a data perspective, and I am getting handed a sandwich of sorts and not a good one!”.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Towards Data Science

APRIL 6, 2023

Principles, practices, and examples for ensuring high quality data flows Source: DreamStudio (generated by author) Nearly 100% of companies today rely on data to power business opportunities and 76% use data as an integral part of forming a business strategy. PagerDuty), and — if you have one — a data catalog (e.g.,

Unstructured Data 98

Unstructured Data Data Metadata Coding 98

Monte Carlo

MARCH 7, 2024

Why prompt engineering isn’t all that and a bag of SQL queries Understand vector databases Create AI differentiation with RAG Find and solve real business problems High-quality data always lives up to the hype What is prompt engineering? Table of Contents Why is prompt engineering important?

Engineering 59

Engineering Database High Quality Data Data Engineering 59

Databand.ai

AUGUST 30, 2023

Data Consistency vs Data Integrity: Similarities and Differences Joseph Arnold August 30, 2023 What Is Data Consistency? Data consistency refers to the state of data in which all copies or instances are the same across all systems and databases.

Data Integration 52

Data Integration Data Cleanse Data Validation High Quality Data 52

Monte Carlo

APRIL 1, 2024

Implement Routine Data Audits Build a data cleaning cadence into your data teams’ schedule. Routine data quality checks will not only help to reduce the risk of discrepancies in your data, but it will also help to fortify a culture of high-quality data throughout your organization.

High Quality Data 52

High Quality Data Datasets Data Data Pipeline 52