Data Engineering Podcast

DECEMBER 31, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. What is in store for the future of Pravega?

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Data Engineering Podcast

NOVEMBER 27, 2022

In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution. Can you describe what you are building at Voltron Data and the story behind it? Can you describe what you are building at Voltron Data and the story behind it?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Edureka

AUGUST 2, 2023

Raw data, however, is frequently disorganised, unstructured, and challenging to work with directly. Data processing analysts can be useful in this situation. Let’s take a deep dive into the subject and look at what we’re about to study in this blog: Table of Contents What Is Data Processing Analysis?

Data Process 52

Data Process Process Data Cleanse Data Mining 52

Precisely

JULY 25, 2023

Organizations that run SAP can use Excel-to-SAP automation to do more with less, while also increasing agility and improving their SAP master data management process automation. Automate Evolve is designed to digitize a specific class of processes in which process and data are decidedly interdependent.

Data Process 52

Data Process Process Data Data Integration 52

LinkedIn Engineering

JULY 19, 2023

We’ll also introduce OpenHouse’s control plane, specifics of the deployed system at LinkedIn including our managed Iceberg lakehouse, and the impact and roadmap for future development of OpenHouse, including a path to open source. Managed Iceberg Lakehouse At LinkedIn, OpenHouse tables are persisted on HDFS in Iceberg table format.

Big Data 78

Big Data Data Management Management Metadata 78

Edureka

APRIL 22, 2025

AI-powered data engineering solutions make it easier to streamline the data management process, which helps businesses find useful insights with little to no manual work. Real-time data processing has emerged The demand for real-time data handling is expected to increase significantly in the coming years.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Snowflake

NOVEMBER 19, 2024

To overcome these hurdles, CTC moved its processing off of managed Spark and onto Snowflake, where it had already built its data foundation. Thanks to the reduction in costs, CTC now maximizes data to further innovate and increase its market-making capabilities.

Data Governance 98

Data Governance Government Healthcare Management 98

Ascend.io

DECEMBER 4, 2024

Examples include “reduce data processing time by 30%” or “minimize manual data entry errors by 50%.” Deploy DataOps DataOps , or Data Operations, is an approach that applies the principles of DevOps to data management. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Snowflake

MARCH 16, 2023

Choosing Snowflake also benefited John Lewis’ data scientists. By splitting the platform’s compute and storage capabilities, the team now has access to all the data management tools they need without the risk of racking up huge costs. In addition, Dynamic Data Masking ensures the safety of John Lewis’ data.

Data Process 52

Data Process Cloud Process IT 52

Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. Data observability has been gaining adoption for a number of years now, with a large focus on data warehouses.

Process 147

Process Data Lake High Quality Data Machine Learning 147

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Management Data Lake Data Warehouse 52

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Can you describe what you mean by a "composable CDP"? What are some of the key ways that it differs from the ways that we think of a CDP today?

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing. See it in action and schedule a demo with one of our data experts today.

Data Management 52

Data Management Management Data Lake Data Governance 52

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing. See it in action and schedule a demo with one of our data experts today.

Data Management 52

Data Management Management Data Lake Data Governance 52

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing.

Data Management 52

Data Management Management Data Lake Data Governance 52

Ascend.io

DECEMBER 4, 2024

Examples include “reduce data processing time by 30%” or “minimize manual data entry errors by 50%.” Deploy DataOps DataOps , or Data Operations, is an approach that applies the principles of DevOps to data management. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

AltexSoft

JULY 18, 2023

It allows data scientists to analyze large datasets and interactively run jobs on them from the R shell. Big data processing. Despite these nuances, Spark’s high-speed processing capabilities make it an attractive choice for big data processing tasks. Here are some of the possible use cases.

Big Data 52

Big Data Data Process Process Hadoop 52

Cloudera

DECEMBER 17, 2024

Since 5G networks began rolling out commercially in 2019, telecom carriers have faced a wide range of new challenges: managing high-velocity workloads, reducing infrastructure costs, and adopting AI and automation. From customer service to network management, AI-driven automation will transform the way carriers run their businesses.

Building 62

Building Telecommunication Data Architecture Architecture 62

DataKitchen

MARCH 20, 2025

Understanding this framework offers valuable insights into team efficiency, operational excellence, and data quality. Process-centric data teams focus their energies predominantly on orchestrating and automating workflows. The path to better data management is accessible and rewarding, regardless of your starting point.

Pipeline-centric 65

Pipeline-centric Database-centric Process Data 65

Snowflake

JULY 18, 2023

At the heart of anecdotes’ platform sits a data pipeline leveraging Snowflake’s data sharing capabilities. This ensures every environment correlates with company standards, easily supports connected applications, and offers enhanced security and data management. The Data Cloud unlocks massive go-to-market opportunities.”

Data Process 52

Data Process Process Data Lake Data Ingestion 52

Snowflake

NOVEMBER 11, 2024

Internally, banks are using AI to reduce the burden of data management, including data lineage and data quality controls, or drive efficiencies with business intelligence particularly in call centers. Commercially, we heard AI use cases around treasury services, fraud detection and risk analytics.

Banking 96

Banking Finance Retail Food 96

Snowflake

APRIL 9, 2025

It employs Snowpark Container Services to build scalable AI/ML models for satellite data processing and Snowflake AI/ML functions to enable advanced analytics and predictive insights for satellite operators.

Pharmaceutical 97

Pharmaceutical Manufacturing Data Ingestion SQL 97

Ascend.io

OCTOBER 28, 2024

Advanced Data Transformation Techniques For data engineers ready to push the boundaries, advanced data transformation techniques offer the tools to tackle complex data challenges and drive innovation. Automated testing and validation steps can also streamline transformation processes, ensuring reliable outcomes.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Data Engineering Podcast

JANUARY 30, 2022

He recently wrote a book on effective patterns for Pandas code, and in this episode he shares advice on how to write efficient data processing routines that will scale with your data volumes, while being understandable and maintainable. What are the main tasks that you have seen Pandas used for in a data engineering context?

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Snowflake

APRIL 7, 2025

The Snowflake Native App Framework enables us to develop and deploy data-intensive applications directly within the Snowflake ecosystem. This integration allows us to leverage Snowflake's robust data processing and storage features, enabling our AI-driven compliance and quality management tools to operate efficiently and at scale.

Food 77

Food Data Transparency Software Engineer Software Engineering 77

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. What was your decision process for building Dozer as open source?

Building 130

Building Machine Learning SQL Python 130

Precisely

MARCH 3, 2025

And many such customers are enabling their business owners or data stewards who are closest to the data and processes as citizen developers of automation solutions for those business areas. For example, SAP ERP master data processes are complex and often highly data intensive.

Process 58

Process Government Data Finance 58

phData: Data Engineering

FEBRUARY 21, 2025

A data warehouse acts as a single source of truth for an organization’s data, providing a unified view of its operations and enabling data-driven decision-making. A data warehouse enables advanced analytics, reporting, and business intelligence. Data integrations and pipelines can also impact latency.

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Data Engineering Weekly

APRIL 24, 2025

. - Leveraging Cloud Capabilities: He noted recent developments in cloud-based infrastructure as significant enablers, providing scalable and sophisticated tools for data management and model deployment.

Government 52

Government Data Governance High Quality Data Machine Learning 52

Precisely

NOVEMBER 21, 2024

This relationship is particularly important in SAP® environments, where data and processes must work together seamlessly at scale. To achieve true transformation, you need an aligned approach where both processes and data management evolve together.

Data Integration 58

Data Integration High Quality Data Data Manufacturing 58

Cloudera

OCTOBER 7, 2021

The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., need to integrate multiple “point solutions” used in a data ecosystem) and organization reasons (e.g., difficulty to achieve cross-organizational governance model).

Architecture 124

Architecture Metadata Kafka Government 124

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. Can you start by describing what the Jepsen project is?

Systems 100

Systems Building Scala Java 100

Snowflake

NOVEMBER 29, 2023

The data is there, it’s just not FAIR: Findable, Accessible, Interoperable and Reusable. Defining FAIR data and it’s applications for life sciences FAIR was a term coined in 2016 to help define good data management practices within the scientific realm. The principles emphasize machine-actionability (i.e.,

Metadata 107

Metadata Healthcare Medical Data Storage 107

Data Engineering Podcast

JANUARY 6, 2019

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. How does it fit into the Hadoop ecosystem?

Data Analytics 100

Data Analytics Hadoop Kafka Media 100

Data Engineering Podcast

MAY 6, 2021

Summary Spark is one of the most well-known frameworks for data processing, whether for batch or streaming, ETL or ML, and at any scale. In this episode Jean-Yves Stephan shares the work that he is doing at Data Mechanics to make it sing on Kubernetes. What do you have planned for the future of the platform?

Cloud 100

Cloud Data Warehouse Data Engineer Data Engineering 100