Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. Data lakes are notoriously complex. Data lakes are notoriously complex.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Cloudyard

FEBRUARY 19, 2024

The data, originating from different formats and sources, requires consolidation into Snowflake tables for comprehensive analysis. Therefore, Snowpark, with its capabilities in simplifying complex data workflows, becomes instrumental in achieving this objective. The journey begins with customer invoice data stored in a CSV file.

Data Process 40

Data Process Process Data Workflow Data 40

Snowflake

JUNE 6, 2024

Schedule data ingestion, processing, model training and insight generation to enhance efficiency and consistency in your data processes. That’s why we partner with Hex , to provide data teams with access to best-in-class tools to drive more decisions for the business and more value from their data.

SQL 115

SQL Python Machine Learning Data Workflow 115

Data Engineering Weekly

FEBRUARY 2, 2025

I finally found a good critique that discusses its flaws, such as multi-hop architecture, inefficiencies, high costs, and difficulties maintaining data quality and reusability. The article advocates for a "shift left" approach to data processing, improving data accessibility, quality, and efficiency for operational and analytical use cases.

Data Engineering 57

Data Engineering Data Engineer Engineering Data Lake 57

Data Engineering Podcast

JANUARY 30, 2022

Matt Harrison is a Python expert with a long history of working with data who now spends his time on consulting and training. What are some of the utility features that you have found most helpful for data processing? Pandas is a tool that spans data processing and data science.

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Data Engineering Podcast

AUGUST 28, 2022

What are the different concerns that need to be included in a stack that supports fully automated data workflows? There was recently an interesting article suggesting that the "left-to-right" approach to data workflows is backwards.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Engineering at Meta

FEBRUARY 4, 2025

To prevent this issue, we built verification in the post-processing stage to ensure that the user ID column in the data matches the identifier for the user whose logs we are generating.

Accessibility 80

Accessibility Accessible Raw Data Data Warehouse 80

Edureka

APRIL 22, 2025

Since all of Fabric’s tools run natively on OneLake, real-time performance without data duplication is possible in Direct Lake mode. Because of the architecture’s ability to abstract infrastructure complexity, users can focus solely on data workflows.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Netflix Tech

NOVEMBER 14, 2023

In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. Stay tuned for a new post on this!

Metadata 101

Metadata Data Pipeline Scala Data Process 101

DataKitchen

JANUARY 25, 2022

DataOps improves the robustness, transparency and efficiency of data workflows through automation. For example, DataOps can be used to automate data integration. Previously, the consulting team had been using a patchwork of ETL to consolidate data from disparate sources into a data lake.

Consulting 97

Consulting Recruitment Data Lake Data Engineering 97

Ascend.io

JULY 3, 2024

This methodology emphasizes automation, collaboration, and continuous improvement, ensuring faster, more reliable data workflows. With data workflows growing in scale and complexity, data teams often struggle to keep up with the increasing volume, variety, and velocity of data. Let’s dive in!

Data Workflow 52

Data Workflow Data Pipeline Data Data Engineering 52

Hevo

SEPTEMBER 1, 2024

Managing and orchestrating data workflows efficiently is crucial in today’s data-driven world. As the amount of data constantly increases with each passing day, so does the complexity of the pipelines handling such data processes.

Data Workflow 52

Data Workflow Data Pipeline Data Data Process 52

Hevo

AUGUST 16, 2024

This blog explores the world of open source data orchestration tools, highlighting their importance in managing and automating complex data workflows. From Apache Airflow to Google Cloud Composer, we’ll walk you through ten powerful tools to streamline your data processes, enhance efficiency, and scale your growing needs.

Google Cloud 52

Google Cloud Data Workflow Data Data Engineering 52

Ascend.io

JULY 3, 2024

Start small, then scale With data workflows growing in scale and complexity, data teams often struggle to keep up with the increasing volume, variety, and velocity of data. This is where DataOps comes ina methodology designed to streamline and automate data workflows, ensuring faster and more reliable data delivery.

Data Workflow 52

Data Workflow Data Pipeline Data Data Engineering 52

Snowflake

JULY 22, 2024

From there, you can address more complex use cases, such as creating a 360-degree view of customers by integrating systems across CRM, ERP, marketing applications, social media handles and other data sources.

Government 119

Government Data Cloud Hadoop 119

Ascend.io

SEPTEMBER 6, 2024

AI-driven data quality workflows deploy machine learning to automate data cleansing, detect anomalies, and validate data. Integrating AI into data workflows ensures reliable data and enables smarter business decisions. Data quality is the backbone of successful data engineering projects.

Data Cleanse 52

Data Cleanse Data Workflow Data Pipeline Machine Learning 52

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. Accelerated Data Analytics DataOps tools help automate and streamline various data processes, leading to faster and more efficient data analytics.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Striim

SEPTEMBER 11, 2024

Furthermore, Striim also supports real-time data replication and real-time analytics, which are both crucial for your organization to maintain up-to-date insights. By efficiently handling data ingestion, this component sets the stage for effective data processing and analysis. Are we using all the data or just a subset?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

AltexSoft

JUNE 7, 2021

Hadoop and Spark are the two most popular platforms for Big Data processing. They both enable you to deal with huge collections of data no matter its format — from Excel tables to user feedback on websites to images and video files. Obviously, Big Data processing involves hundreds of computing units.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. It automates and optimizes data processes, reducing manual effort and the likelihood of errors. But let’s step back for a second.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Cloudyard

JUNE 11, 2024

Read Time: 1 Minute, 48 Second RETRY LAST: In modern data workflows, tasks are often interdependent, forming complex task chains. Ensuring the reliability and resilience of these workflows is critical, especially when dealing with production data pipelines.

Raw Data 52

Raw Data Aggregated Data Data Pipeline Data Workflow 52

Data Engineering Weekly

SEPTEMBER 29, 2024

The article details how bypassing intermediate storage steps reduces latency and improves data processing speed. The approach highlights the importance of streamlining data workflows for faster machine learning model training and deployment.

Data Engineering 62

Data Engineering Data Engineer Engineering SQL 62

Workfall

JULY 18, 2023

Additionally, they offer a way to keep your transformed data up to date without redoing the entire transformation process. By using DBT materializations, you can streamline your data workflows and focus on gaining insights rather than spending excessive time on repetitive calculations.

Datasets 77

Datasets Entertainment Data Data Workflow 77

DataKitchen

APRIL 13, 2021

Composable Analytics — A DataOps Enterprise Platform with built-in services for data orchestration, automation, and analytics. Reflow — A system for incremental data processing in the cloud. Dagster / ElementL — A data orchestrator for machine learning, analytics, and ETL. .

Consulting 120

Consulting Machine Learning Data Science Data Pipeline 120

Netflix Tech

NOVEMBER 20, 2023

IPS provides the incremental processing support with data accuracy, data freshness, and backfill for users and addresses many of the challenges in workflows. IPS enables users to continue to use the data processing patterns with minimal changes. Note that the backfill support is skipped in this blog.

Process 87

Process Data Pipeline Datasets SQL 87

Meltano

OCTOBER 5, 2022

In the same way, a DataOps engineer designs the data assembly line that enables data scientists to derive insights from data analytics faster and with fewer errors. DataOps engineers improve the speed and quality of the data development process by applying DevOps principles to data workflow, known as DataOps.

Engineering 52

Engineering Raw Data Data Pipeline Data Warehouse 52

Data Engineering Weekly

DECEMBER 25, 2023

These technologies are increasingly automating processes like ETL, improving data quality management, and evolving the landscape of data tools. Integrating AI into data workflows is not just a trend but a paradigm shift, making data processes more efficient and intelligent.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Monte Carlo

JULY 9, 2024

Overcoming Challenges to Achieve Data Timeliness Despite the clear benefits of timely data, organizations often encounter several hurdles: Complex Data Pipelines: Intricate data pipelines that involve multiple stages and dependencies can hinder smooth data flow.

Food 52

Food Data Pipeline Data Data Workflow 52

Data Engineering Weekly

DECEMBER 16, 2024

Evolution of Data Lake Technologies The data lake ecosystem has matured significantly in 2024, particularly in table formats and storage technologies. Query Optimization and Cost Attribution By optimizing their most expensive pipelines , Medium's engineering team demonstrated significant cost savings in their Snowflake environment.

Data Engineering 62

Data Engineering Data Engineer Engineering Unstructured Data 62

Ascend.io

DECEMBER 19, 2023

Snowflake’s Data Marketplace : Enriches data pipelines with external data sources, providing access to a diverse range of datasets and services that can be seamlessly integrated into your analytics and data processing workflows. that you can combine to create custom data workflows.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Monte Carlo

APRIL 28, 2023

DuckDB’s parallel execution capabilities can help DBAs improve the performance of data processing tasks. Researchers : Academics and researchers working with large volumes of data use DuckDB to process and analyze their data more efficiently. What makes DuckDB different?

Data Warehouse 52

Data Warehouse Programming Language Database Datasets 52

Cloudera

AUGUST 21, 2020

Can you talk about some of the technology that helps make managing live streaming data possible? Cloudera DataFlow offers the capability for Edge to cloud streaming data processing. This type of end-to-end data processing that starts at the Edge and ends in the cloud is made possible by using Apache NiFi.

Banking 62

Banking Kafka Cloud Storage Government 62

Knowledge Hut

SEPTEMBER 25, 2023

As an Azure Data Engineer, you will be expected to design, implement, and manage data solutions on the Microsoft Azure cloud platform. You will be in charge of creating and maintaining data pipelines, data storage solutions, data processing, and data integration to enable data-driven decision-making inside a company.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Ascend.io

JULY 11, 2024

It enhances data quality, governance, and optimization, making data retrieval more efficient and enabling powerful automation in data engineering processes. As practitioners using metadata to fuel data teams, we at Ascend understand the critical role it plays in organizing, managing, and optimizing data workflows.

Metadata 52

Metadata IT Government High Quality Data 52

Ascend.io

JUNE 10, 2024

At Ascend, we recognize the immense potential in integrating these advanced AI capabilities into our platform, enabling smarter applications and more efficient data workflows. Snowflake’s investment in expanding data engineering capabilities is a game-changer.

Data Engineering 52

Data Engineering Data Engineer Engineering High Quality Data 52

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Mastering Data Production (#3) Introduction Managing the production phase of data analytics is a daunting challenge. Overseeing multi-tool, multi-dataset, and multi-hop data processes ensures high-quality outputs.

Raw Data 49

Raw Data Data Ingestion Datasets Data 49