Data Engineering Weekly

JANUARY 15, 2025

As large language models (LLMs) and AI agents become indispensable in everything from customer service to autonomous vehicles, the ability to manage, analyze, and optimize unstructured data has become a strategic imperative. Billions of social media posts, hours of video content, and terabytes of sensor data are produced daily.

Data Engineering 62

Data Engineering Data Engineer Unstructured Data Engineering 62

Snowflake

MARCH 26, 2025

This means enterprises can run unstructured data workflows, powered by AI agents, without moving data out of Snowflake which enhances trust and helps support compliance. First, Snowflake has enabled us to strengthen user trust in our app. Second, were optimizing scalability.

Unstructured Data 98

Unstructured Data SQL Data Data Workflow 98

KDnuggets

SEPTEMBER 9, 2024

Exploring the not-so-famous data science libraries that can be useful in your data workflow.

Data Science 145

Data Science Python Data Workflow Data 145

KDnuggets

APRIL 16, 2024

Bring the latest AI implementation to Pandas to improve your data workflow.

Utilities 157

Utilities Data Analysis Data Workflow Data 157

Cloudera

DECEMBER 4, 2024

Our joint collaboration will enable the following: Seamless Data Sharing and Interoperability : The integration enables AWS customers to leverage Cloudera’s data lakehouse capabilities alongside Snowflake’s AI Data Cloud, facilitating unified data access and sharing across platforms Enhanced AI/ML Performance : The partnership optimizes data workflows (..)

AWS 89

AWS Raw Data Relational Database Government 89

Data Engineering Podcast

APRIL 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

FEBRUARY 18, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

KDnuggets

JUNE 17, 2024

Exploring the available text Python packages for your data workflow.

Python 131

Python Data Workflow Data 131

ProjectPro

JUNE 6, 2025

Dagster vs Airflow: Overview Dagster and Airflow are two popular open-source tools that have emerged as leaders in data orchestration. They are often compared because of their shared goal of automating data workflows and widespread adoption in the data engineering community. What is Airflow? What is Dagster?

Pipeline-centric 40

Pipeline-centric Database-centric Data Workflow Data Pipeline 40

Snowflake

MARCH 24, 2025

DataOps.live keeps users at the forefront of data engineering DataOps.live works together with Snowflake to augment and extend native Snowflake features, resulting in advanced DataOps workflows for Snowflake customers. Snowflake and DataOps.lives integrated solutions simplify the development, testing and deployment of data workflows.

Cloud 71

Cloud Data Pipeline Data Workflow Data Engineering 71

ProjectPro

JUNE 6, 2025

Open Source Data Pipeline Tools Open-source data pipeline tools are pivotal in data engineering, offering organizations flexible and scalable solutions for managing the end-to-end data workflow. Here is the list of robust Data Pipeline Tools in Azure for scalable and optimized management of diverse data sources.

Data Pipeline 49

Data Pipeline Google Cloud AWS Kafka 49

Data Engineering Podcast

DECEMBER 24, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

KDnuggets

OCTOBER 22, 2024

Utilize these transformation techniques in your data workflow.

Data Workflow 123

Data Workflow Utilities Data 123

KDnuggets

JANUARY 27, 2025

Master cleaner, faster code with these essential techniques to supercharge your data workflows.

Python 123

Python Data Workflow Data Coding 123

KDnuggets

NOVEMBER 30, 2023

Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.

Data Engineering 136

Data Engineering Data Engineer Engineering Data Workflow 136

ProjectPro

JUNE 6, 2025

It's like the ultimate solution for managing and automating big data workflows. Did you know 93% of seasoned Airflow users are willing to recommend this powerful data orchestration tool. Businesses from various sectors leverage it to manage and automate massive data workflows seamlessly. Crazy, right? stars and 13.4k

PostgreSQL 40

PostgreSQL Metadata MySQL Data Workflow 40

Ascend.io

DECEMBER 4, 2024

Deploy DataOps DataOps , or Data Operations, is an approach that applies the principles of DevOps to data management. It aims to streamline and automate data workflows, enhance collaboration and improve the agility of data teams. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Podcast

JUNE 23, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

ProjectPro

JUNE 6, 2025

Data engineers gain insights into pipeline performance, data movement, and potential bottlenecks. This skill is crucial for maintaining smooth data workflows and ensuring data integrity. This phase also underscores the seamless integration of Azure Data Factory with a range of Azure services.

Data Lake 40

Data Lake Cloud Computing Data Workflow Data Pipeline 40

KDnuggets

JANUARY 4, 2023

These Python packages would improve your data workflow.

Python 122

Python Data Workflow Data Data Science 122

Data Engineering Podcast

JUNE 16, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. Your first 30 days are free! Your first 30 days are free!

Project 130

Project Data Lake High Quality Data Data Workflow 130

ProjectPro

JUNE 6, 2025

However, creating and deploying these agents often involves challenges such as managing complex data workflows, integrating machine learning models, and ensuring scalability across operations. Phidata offers better support for integration with external data sources, whereas CrewAI focuses on refining AI pipelines within its ecosystem.

Building 40

Building Data Workflow Python Data Pipeline 40

Data Engineering Podcast

APRIL 21, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Weekly

NOVEMBER 24, 2024

By creating custom linting rules tailored to their team's needs, Next Insurance has improved its data workflows' maintainability, scalability, and quality, making it easier for engineers to collaborate and debug issues.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

KDnuggets

DECEMBER 6, 2023

This week on KDnuggets: Discover GitHub repositories from machine learning courses, bootcamps, books, tools, interview questions, cheat sheets, MLOps platforms, and more to master ML and secure your dream job • Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company • And much, (..)

Machine Learning 119

Machine Learning Data Engineering Data Engineer Engineering 119

Data Engineering Podcast

DECEMBER 3, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues for every part of your data workflow, from migration to deployment. Datafold has recently launched a 3-in-1 product experience to support accelerated data migrations. Datafold : ![Datafold]([link]

Systems 130

Systems Designing Data Lake SQL 130

Ascend.io

DECEMBER 4, 2024

Deploy DataOps DataOps , or Data Operations, is an approach that applies the principles of DevOps to data management. It aims to streamline and automate data workflows, enhance collaboration and improve the agility of data teams. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Podcast

JUNE 30, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

databricks

JUNE 11, 2025

Deeply integrated with the lakehouse, Lakebase simplifies operational data workflows. It eliminates fragile ETL pipelines and complex infrastructure, enabling teams to move faster and deliver intelligent applications on a unified data platform In this blog, we propose a new architecture for OLTP databases called a lakebase.

Entertainment 81

Entertainment Data Lake Manufacturing Retail 81

RudderStack

JUNE 6, 2025

Apache Airflow addresses the need for a robust, scalable, and flexible solution for orchestrating data workflows. What is Apache Airflow?

Data Workflow 40

Data Workflow Data 40

Data Engineering Podcast

FEBRUARY 4, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Kafka 173

Ascend.io

OCTOBER 28, 2024

Read More: Snowflake Snowpark: Overview, Benefits, and How to Harness Its Power Best Practices in Data Transformation Implementing best practices in data transformation is essential to maintain high-quality, consistent, and secure data workflows.

Raw Data 52

Raw Data Aggregated Data Data Pipeline Data Validation 52

Data Engineering Podcast

SEPTEMBER 17, 2023

Can you describe the workflow for building autonomous linkages across data assets that are modelled as JSON-LD? What are the most interesting, innovative, or unexpected ways that you have seen JSON-LD used for data workflows? When is JSON-LD the wrong choice? When is JSON-LD the wrong choice?

Building 189

Building SQL BI Python 189

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. As you can see, this is in the code part where you are building your data pipelines, a misnomer because this is an over simplification. The other benefit is you can also use parameters and build a generic workflows to be re-used.

Technology 130

Technology Architecture Google Cloud Metadata 130