Data Engineering Podcast

NOVEMBER 5, 2023

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Striim

APRIL 18, 2025

Many organizations struggle with: Inconsistent data formats : Different systems store data in varied structures, requiring extensive preprocessing before analysis. Siloed storage : Critical business data is often locked away in disconnected databases, preventing a unified view. Start Your Free Trial | Schedule a Demo

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

MAY 12, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

JUNE 9, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

JUNE 2, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst : ![Starburst

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

NOVEMBER 19, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. You shouldn't have to throw away the database to build with fast-changing data.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. With Materialize, you can!

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

DECEMBER 3, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. SQL Server version upgrade) Section 2: Types of Migrations for Infrastructure Focus Storage migration: Moving data between systems (HDD to SSD, SAN to NAS, etc.) Starburst : ![Starburst

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

DECEMBER 17, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication.

Project 130

Project Data Lake High Quality Data Data Workflow 130

ProjectPro

JUNE 6, 2025

What is Data Quality, and Why is it Important? Data Quality refers to the degree to which data is accurate, reliable, consistent, and relevant for its intended purpose. High-quality data is essential for organizations to derive meaningful insights, make informed decisions, and meet regulatory requirements.

Building 40

Building High Quality Data Datasets Hadoop 40

Monte Carlo

OCTOBER 31, 2024

The foundational skills are similar between traditional data engineers and AI data engineers are similar, with AI data engineers more heavily focused on machine learning data infrastructure, AI-specific tools, vector databases, and LLM pipelines. Let’s dive into the tools necessary to become an AI data engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

DataKitchen

MARCH 20, 2025

Process-centric data teams focus their energies predominantly on orchestrating and automating workflows. They have demonstrated that robust, well-managed data processing pipelines inevitably yield reliable, high-quality data.

Pipeline-centric 65

Pipeline-centric Database-centric Process Data 65

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period. Both integrations will be available early 2024.

Kafka 64

Kafka Database High Quality Data Cloud 64

ProjectPro

JUNE 6, 2025

Data engineers are the ones who are responsible for ingesting raw data from multiple sources and processing it to serve clean datasets to Data Scientists and Data Analysts so they can run machine learning models and data analytics, respectively. Destination refers to a landing area where the data is taken to.

Data Ingestion 45

Data Ingestion Data Engineering Data Engineer Project 45

ProjectPro

JUNE 6, 2025

It is crucial to have the data in a design that supports the application, which puts it in motion and provides meaningful information while the data is at rest. Data modeling is essential because it enables businesses to visualize these operations and design, build, and deploy high-quality data assets.

Data Engineering 40

Data Engineering Data Engineer Project Engineering 40

ProjectPro

JUNE 6, 2025

TensorFlow) Strong communication and presentation skills Data Scientist Salary According to the Payscale, Data Scientists earn an average of $97,680. Employ automated techniques to extract data from primary and secondary data sources Analyze data and present it in the form of graphs and reports.

Data Science 56

Data Science Data Mining Data Architect BI 56

ProjectPro

JUNE 6, 2025

Whereas data architects focus on data extraction, transformation, and loading data, they consider how it should be structured and arranged. Data engineers and architects can provide high-quality data useful for executive decisions. Data Engineer vs Data Architect - Who Does What?

Data Architect 40

Data Architect Data Engineering Data Engineer Building 40

ProjectPro

JUNE 6, 2025

Step 1: Collecting and Preparing Data The first step in any AI project, including generative AI , is gathering and preparing high-quality data. The quality of the data significantly impacts the performance of your model and the quality of AI generated content. books, articles) and image datasets (e.g.,

Building 52

Building Banking SQL Deep Learning 52

Precisely

APRIL 7, 2025

Use case (Retail): As an example, imagine a retail company has a customer database with names and addresses, but many records are missing full address information. The solution: They use a data appending process to match their existing data with a third-party database that contains full street addresses. Plan for it.

Retail 75

Retail Datasets Data Portfolio 75

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis. The team at Audio Analytic are working to impart a sense of hearing to our myriad devices with their sound recognition technology.

Data Collection 100

Data Collection Management High Quality Data Metadata 100

U-Next

FEBRUARY 27, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

ProjectPro

JUNE 6, 2025

Schema Enforcement and Evolution Delta lake will enforce schema when writing data to the storage. Thus, columns and their data types are maintained, preventing data corruption and achieving data reliability and high-quality data. This data will be available downstream for analytics and reporting.

Data Lake 63

Data Lake Data Warehouse Metadata Unstructured Data 63

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were. Those systems have been taught to normalize the data for storage on their own.

Data Engineering 130

Data Engineering Data Engineer Engineering Database-centric 130

U-Next

MARCH 7, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

Precisely

JULY 12, 2024

However, simply having high-quality data does not, of itself, ensure that an organization will find it useful. Data observability: P revent business disruption and costly downstream data and analytics issues using intelligent technology that proactively alerts you to data anomalies and outliers.

Data Integration 105

Data Integration Data Data Governance Datasets 105

Databand.ai

MAY 30, 2023

Data quality refers to the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context. High-quality data is essential for making well-informed decisions, performing accurate analyses, and developing effective strategies.

Data Cleanse 85

Data Cleanse Data Governance Data Validation High Quality Data 85

ProjectPro

JUNE 6, 2025

This function also allows DBT to automatically handle schema changes and cross-database functionality, improving manageability within your project. This file also enables project-wide settings for database connections, materialization strategies, and configurations, allowing for uniformity and streamlined changes across the project.

Raw Data 40

Raw Data SQL Datasets Data Pipeline 40

Towards Data Science

MARCH 29, 2023

Great reads on modeling, processes, and leadership Photo by Emil Widlund on Unsplash At the very start of my journey in data, I thought I was going to be a data scientist, and my first foray into data was centered on studying statistics and linear algebra, not software engineering or database management.

Data Warehouse 96

Data Warehouse BI Healthcare Database 96