Data Engineering Podcast

OCTOBER 29, 2023

Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. With Materialize, you can!

Database 189

Database SQL BI Machine Learning 189

Seattle Data Guy

APRIL 15, 2025

Planning out your data infrastructure in 2025 can feel wildly different than it did even five years ago. Everyone is talking about AI, chatbots, LLMs, vector databases, and whether your data stack is “AI-ready.” The ecosystem is louder, flashier, and more fragmented.

Database 182

Database Data IT Big Data 182

Christophe Blefari

JANUARY 11, 2025

The Data News are here to stay, the format might vary during the year, but here we are for another year. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. HNY 2025 ( credits ) Happy new year ✨ I wish you the best for 2025. Not really digest.

Data 130

Data Data Warehouse Coding Programming Language 130

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Analytics Vidhya

FEBRUARY 22, 2023

Introduction Data replication is also known as database replication, which is copying data to ensure that all information remains consistent across all data resources in real-time. data replication is like a safety net that keeps your information safe from disappearing or falling through the cracks.

Database 276

Database Data NoSQL Datasets 276

Data Engineering Podcast

APRIL 14, 2024

Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Simon Späti

MARCH 28, 2025

The database landscape has reached 394 ranked systems across multiple categoriesrelational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases. As AI applications multiply quickly, vector technologies have become a frontier that data engineers must explore.

Technology 130

Technology PostgreSQL MySQL Database 130

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake. Striim automates the rest.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

KDnuggets

OCTOBER 18, 2023

Cloud databases have made it easier and cheaper to develop enterprise-level applications, offering flexibility, convenience, and standard database functionality. See what KDnuggets recommends.

Database 141

Database Cloud IT Data Engineering 141

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. In this episode he explains how he used the combination of Apache Arrow, Flight, Datafusion, and Parquet to lay the foundation of the newest version of his time-series database. Data lakes are notoriously complex.

Database 162

Database Technology Data Lake High Quality Data 162

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 120

Data Warehouse SQL Programming Language Data 120

Analytics Vidhya

FEBRUARY 6, 2023

Especially while working with databases, it is often considered a good practice to follow a design pattern. This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? The pattern is not an actual code but a template that can be used to solve problems in different situations.

Accessibility 239

Accessibility Accessible Python Database 239

KDnuggets

JUNE 28, 2024

Explore the world of modern databases that are fast, secure, and cost-efficient, designed to tackle large-scale and diverse data challenges.

Database 137

Database SQL Designing Data 137

KDnuggets

MARCH 5, 2024

Looking to learn SQL and databases to level up your data science skills? Learn SQL, database internals, and much more with these free university courses.

SQL 144

SQL Database Data Science Data 144

KDnuggets

NOVEMBER 13, 2023

This week, we delve into the vital world of Databases, SQL, Data Management, and Statistical Concepts in Data Science. Welcome back to Week 2 of KDnuggets’ "Back to Basics" series.

Database 133

Database SQL Data Management Management 133

KDnuggets

SEPTEMBER 13, 2024

Learn how to install ClickHouse DBMS, create a database, and run SQL queries using native and Python clients.

Database 123

Database SQL Python Data Engineering 123

Christophe Blefari

MARCH 15, 2024

Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. Pandera, a data validation library for dataframes, now supports Polars.

Metadata 272

Metadata Data Software Engineer Software Engineering 272

Precisely

JANUARY 9, 2025

Does the LLM capture all the relevant data and context required for it to deliver useful insights? Not to mention the crazy stories about Gen AI making up answers without the data to back it up!) Are we allowed to use all the data, or are there copyright or privacy concerns? But simply moving the data wasnt enough.

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

Monte Carlo

DECEMBER 18, 2024

As we turn the corner into 2025, were excited to announce that for the 7th quarter in a row, Monte Carlo has been named G2s #1 Data Observability Platform, as well as #1 in the Data Quality category. Knowing our products are helping our customers achieve their data goals means everything to us. Image courtesy of G2.

Database 93

Database High Quality Data Data Software Engineer 93

Towards Data Science

JANUARY 30, 2025

Building more efficient AI TLDR : Data-centric AI can create more efficient and accurate models. I experimented with data pruning on MNIST to classify handwritten digits. What if I told you that using just 50% of your training data could achieve better results than using the fulldataset? Image byauthor.

Database-centric 106

Database-centric Datasets Data Architecture 106

Monte Carlo

NOVEMBER 26, 2024

Here’s where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024—plus a few predictions of my own. 2025 data engineering trends incoming. Small data is the future of AI (Tomasz) 7. The lines are blurring for analysts and data engineers (Barr) 8. Table of Contents 1.

Data Engineering 105

Data Engineering Data Engineer Engineering Unstructured Data 105

Analytics Vidhya

JUNE 24, 2023

Introduction In the era of data-driven decision-making, having accurate data modeling tools is essential for businesses aiming to stay competitive. As a new developer, a robust data modeling foundation is crucial for effectively working with databases.

Database 218

Database Utilities Data Data Science 218

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 95

Data Lake Cloud Storage Metadata Data Warehouse 95

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. With Materialize, you can!

BI 208

BI SQL Machine Learning Data 208

Towards Data Science

DECEMBER 10, 2024

Three Zero-Cost Solutions That Take Hours, NotMonths A data quality certified pipeline. Source: unsplash.com In my career, data quality initiatives have usually meant big changes. Whats more, fixing the data quality issues this way often leads to new problems. Create a custom dashboard for your specific data qualityproblem.

PostgreSQL 75

PostgreSQL Data Python SQL 75

Snowflake

NOVEMBER 26, 2024

Many of our customers — from Marriott to AT&T — start their journey with the Snowflake AI Data Cloud by migrating their data warehousing workloads to the platform. Today we’re focusing on customers who migrated from a cloud data warehouse to Snowflake and some of the benefits they saw. million in cost savings annually.

Data Warehouse 104

Data Warehouse Cloud PostgreSQL Hadoop 104

Snowflake

NOVEMBER 11, 2024

It’s easy these days for an organization’s data infrastructure to begin looking like a maze, with an accumulation of point solutions here and there. Snowflake is committed to doing just that by continually adding features to help our customers simplify how they architect their data infrastructure. Here’s a closer look.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Pinterest Engineering

NOVEMBER 18, 2024

Liang Mou; Staff Software Engineer, Logging Platform | Elizabeth (Vi) Nguyen; Software Engineer I, Logging Platform | In today’s data-driven world, businesses need to process and analyze data in real-time to make informed decisions. What is Change Data Capture? Support highly distributed database setup.

Kafka 50

Kafka MySQL Database Software Engineer 50

The Pragmatic Engineer

OCTOBER 17, 2024

The current database includes 2,000 server types in 130 regions and 340 zones. Storing data: data collected is stored to allow for historical comparisons. Results are stored in git and their database, together with benchmarking metadata. Visualizing the data: the frontend that allows querying of live and historic data.

Cloud 326

Cloud AWS Metadata Cloud Computing 326

databricks

MAY 27, 2025

Until now, sharing data between enterprise systems often meant complex pipelines, duplication, and lock-in. With Oracles support for Delta Sharing, thats no longer the case.

BI 52

BI Database Systems Data 52

Data Engineering Weekly

MAY 25, 2025

Dagster Components is now here Components provides a modular architecture that enables data practitioners to self-serve while maintaining engineering quality. Understanding this fact will help data tools break new ground with the advancement of AI agents. and Lite 2.0) to pinpoint drop-offs and high retention sections.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Jesse Anderson

FEBRUARY 11, 2025

Semih is a researcher and entrepreneur with a background in distributed systems and databases. He then pursued his doctoral studies at Stanford University, delving into the complexities of database systems. Dont forget to subscribe to my YouTube channel to get the latest on Unapologetically Technical!

Computer Science 130

Computer Science Database Design Software Engineer Software Engineering 130

Seattle Data Guy

NOVEMBER 8, 2024

Whether it was moving data from a local database instance to S3 or some other data storage layer. As… Read more The post What Is AWS DMS And Why You Shouldn’t Use It As An ELT appeared first on Seattle Data Guy. It was interesting to see AWS DMS used in this manner. But it’s not what DMS was built for.

AWS 130

AWS IT Data Storage Database 130

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. It is meant to handle massive volumes of data across many commodity servers while maintaining high availability with no single point of failure.

NoSQL 259

NoSQL Database Project Systems 259