Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. No more scripts, just SQL.

Metadata 100

Metadata BI Data Warehouse Government 100

Cloudera

FEBRUARY 8, 2021

Below is the entire set of steps in the data lifecycle, and each step in the lifecycle will be supported by a dedicated blog post(see Fig. 1): Data Collection – data ingestion and monitoring at the edge (whether the edge be industrial sensors or people in a vehicle showroom).

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineer Data Engineering Engineering 98

Monte Carlo

APRIL 28, 2025

Next, look for automatic metadata scanning. This basically means the tool updates itself by pulling in changes to data structures from your systems. It has real-time metadata updates, deep data lineage, and its flexible if you want to customize or extend it for your teams specific needs.

Metadata 64

Metadata Hadoop Data SQL 64

Cloudera

AUGUST 6, 2024

Open data is the future. And for that future to be a reality, data teams must shift their attention to metadata, the new turf war for data. The need for unified metadata While open and distributed architectures offer many benefits, they come with their own set of challenges. A few solutions manage both.

Metadata 82

Metadata Government Datasets Architecture 82

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

FEBRUARY 20, 2022

In this episode Guy Yachdav, director of software engineering for ImmunAI, shares the complexities that are inherent to managing data workflows for bioinformatics. RudderStack’s smart customer data pipeline is warehouse-first. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Ascend.io

DECEMBER 4, 2024

As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Tool sprawl is another hurdle that data teams must overcome.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. This generalisation makes their data models complex and cryptic and require domain expertise. As you do not want to start your development with uncertainty, you decide to go for the operational raw data directly.

Systems 98

Systems Raw Data Metadata Data Cleanse 98

Data Engineering Weekly

NOVEMBER 24, 2024

The system leverages a combination of an event-based storage model in its TimeSeries Abstraction and continuous background aggregation to calculate counts across millions of counters efficiently. link] Grab: Metasense V2 - Enhancing, improving, and productionisation of LLM-powered data governance. Boyter on Bloom Filters and SQLite.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Data Engineering Podcast

JUNE 16, 2024

What are the other systems that feed into and rely on the Trino/Iceberg service? what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? Email hosts@dataengineeringpodcast.com with your story.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Engineering at Meta

FEBRUARY 4, 2025

Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand.

Accessible 85

Accessible Accessibility Raw Data Data Warehouse 85

Towards Data Science

OCTOBER 25, 2023

As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. You probably work closely with the colleague who builds the data pipeline that you were consuming. There was only one data team, two at most. How did they do it?

Data Pipeline 98

Data Pipeline SQL Data Architecture Data 98

Cloudera

MARCH 14, 2023

In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development. Key requirements for building data pipelines Every data pipeline starts with a business requirement.

Data Pipeline 84

Data Pipeline Designing Kafka Metadata 84

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 89

Systems Metadata Media Kafka 89

Data Engineering Podcast

JULY 5, 2021

Summary At the core of every data pipeline is an workflow manager (or several). Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need.

Systems 100

Systems Management Data Warehouse Programming Language 100

Ascend.io

DECEMBER 4, 2024

As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Tool sprawl is another hurdle that data teams must overcome.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Netflix Tech

NOVEMBER 14, 2023

We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Furthermore, we’ll delve into the inner workings of Psyberg, its unique features, and how it integrates into our data pipelining workflows. What is late-arriving data? How does late-arriving data impact us?

Data Engineering 109

Data Engineering Data Engineer Engineering Metadata 109

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. Atlan is the metadata hub for your data ecosystem. What are some of the unique characteristics of that information?

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

AUGUST 13, 2022

Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Metadata 100

Metadata MongoDB MySQL Scala 100

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. It enhances data quality, governance, and automation, transforming raw data into valuable insights. Imagine a library with millions of books but no catalog system to organize them.

Metadata 52

Metadata IT Government High Quality Data 52

Data Engineering Weekly

FEBRUARY 18, 2025

Kafka is designed to be a black box to collect all kinds of data, so Kafka doesn't have built-in schema and schema enforcement; this is the biggest problem when integrating with schematized systems like Lakehouse. It excels in event-driven architectures and data pipelines. Fluss is tailored for real-time analytics.

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Struggling with broken pipelines? Missing data? Atlan is the metadata hub for your data ecosystem.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems. ETL workflows), as well as downstream (e.g.

Systems 95

Systems Media Machine Learning Data Warehouse 95

Snowflake

MAY 15, 2024

There’s a huge gap between the data quality most companies have by default and the data quality needed for successful AI. And that gap is directly affecting the performance and reliability of AI systems everywhere. Metaplane ensures that every company can trust the data that powers their business.

Cloud 126

Cloud Metadata Data Pipeline Government 126

Data Engineering Podcast

DECEMBER 29, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Struggling with broken pipelines? Missing data? Atlan is the metadata hub for your data ecosystem.

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

JULY 17, 2022

Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Weekly

APRIL 20, 2025

[link] Alex Miller: Decomposing Transactional Systems I was re-reading Jack Vanlightly's excellent series on understanding the consistency model of various lakehouse formats when I stumbled upon the blog on decomposing transaction systems. We all know that data freshness plays a critical role in the performance of Lakehouse.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Podcast

OCTOBER 23, 2022

Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. Atlan is the metadata hub for your data ecosystem. Summary Agile methodologies have been adopted by a majority of teams for building software applications.

Project 130

Project Metadata MongoDB MySQL 130

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Our open, interoperable platform is deployed easily in all data ecosystems, and includes unique security and governance capabilities. 4-Ready for modern data fabric architectures.

Database 92

Database Cloud Systems Management 92

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. 60+ speakers from LinkedIn, Shopify, Amazon, Lyft, Grammarly, Mistral, et al.

Data Engineer 62

Data Engineer Data Engineering Engineering Pipeline-centric 62

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Netflix Tech

NOVEMBER 14, 2023

Now, let’s explore the state of our pipelines after incorporating Psyberg. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. The session metadata table can then be read to determine the pipeline input.

Metadata 102

Metadata Data Pipeline Scala Data Process 102

Data Engineering Weekly

FEBRUARY 23, 2025

[link] Alireza Sadeghi: Open Source Data Engineering Landscape 2025 This article comprehensively overviews the 2025 open-source data engineering landscape, highlighting key trends, active projects, and emerging technologies. I wonder if these systems expand more capabilities that eventually fall on their own weight.

Data Engineer 62

Data Engineer Data Engineering Engineering Kafka 62

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineer 106

Data Engineer Data Engineering Engineering Metadata 106