Netflix Tech

NOVEMBER 14, 2023

We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Furthermore, we’ll delve into the inner workings of Psyberg, its unique features, and how it integrates into our data pipelining workflows. It also becomes inefficient as the data scale increases.

Data Engineer 108

Data Engineer Data Engineering Engineering Metadata 108

Cloudera

MARCH 14, 2023

In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development. Key requirements for building data pipelines Every data pipeline starts with a business requirement.

Data Pipeline 84

Data Pipeline Designing Kafka Metadata 84

Data Engineering Weekly

NOVEMBER 3, 2024

The challenges around memory, data size, and runtime are exciting to read. Sampling is an obvious strategy for data size, but the layered approach and dynamic inclusion of dependencies are some key techniques I learned with the case study. Passes include app-brain-date networking, birds of a feature, post-event parties, etc.

Data Engineer 63

Data Engineer Data Engineering Engineering Pipeline-centric 63

Data Engineering Weekly

FEBRUARY 18, 2025

Kafka is designed for streaming events, but Fluss is designed for streaming analytics. Architecture Difference The first difference is the Data Model. It excels in event-driven architectures and data pipelines. It maintains metadata, manages tablet allocation, lists nodes, and handles permissions.

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Data Engineering Podcast

NOVEMBER 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 130

Building IT Metadata MongoDB 130

François Nguyen

JANUARY 18, 2021

In every step,we do not just read, transform and write data, we are also doing that with the metadata. Last part, it was added the data security and privacy part. Every data governance policy about this topic must be read by a code to act in your data platform (access management, masking, etc.)

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. It enhances data quality, governance, and automation, transforming raw data into valuable insights. This is what managing data without metadata feels like. Chaos, right?

Metadata 52

Metadata IT Government High Quality Data 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing. ” He/She is managing triggers, he/she needs to check conditions (event type ?

Technology 130

Technology Architecture Google Cloud Metadata 130

Netflix Tech

NOVEMBER 14, 2023

Now, let’s explore the state of our pipelines after incorporating Psyberg. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. The session metadata table can then be read to determine the pipeline input.

Metadata 101

Metadata Data Pipeline Scala Data Process 101

Data Engineering Podcast

JUNE 17, 2023

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. What are the most interesting, innovative, or unexpected ways that you have seen column-aware data modeling used?

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Data Engineering Weekly

FEBRUARY 23, 2025

Try Astro Free → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA. Let me know in the comments.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Learn more about Iceberg Tables here. Learn more here.

Metadata 118

Metadata Python AWS Government 118

Data Engineering Podcast

NOVEMBER 26, 2023

Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis Logo]([link] Developing event-driven pipelines is going to be a lot easier - Meet Functions! Developing event-driven pipelines is going to be a lot easier - Meet Functions!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Instead, Let’s examine the current data pipeline architecture and ask why data quality is expensive. Instead of looking at the implementation of the data quality frameworks, Let's examine the architectural patterns of the data pipeline.

Engineering 111

Engineering Kafka Data Pipeline Data Warehouse 111

Data Engineering Weekly

APRIL 6, 2025

You won't want to miss this live event on April 23rd! This article highlights their growing complexity, from multimodal interaction to enterprise adoption, underscoring the data and infrastructure challenges beneath the surface. Introducing Apache Airflow® 3.0 Be among the first to see Airflow 3.0

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Cloudera

OCTOBER 7, 2021

Application Logic: Application logic refers to the type of data processing, and can be anything from analytical or operational systems to data pipelines that ingest data inputs, apply transformations based on some business logic and produce data outputs.

Architecture 124

Architecture Metadata Kafka Government 124

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Weekly

MARCH 23, 2025

Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA.

Data Engineer 63

Data Engineer Data Engineering Engineering Data 63

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Machine Learning 100

Data Engineering Podcast

OCTOBER 2, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Metadata 100

Metadata Electronics MongoDB MySQL 100

Data Engineering Podcast

AUGUST 21, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. RudderStack helps you build a customer data platform on your warehouse or data lake.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Engineer 100

Data Engineer Data Engineering MongoDB Metadata 100

Data Engineering Podcast

OCTOBER 28, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo! For someone who wants to get started with Dagster can you describe a typical workflow for writing a data pipeline?

Building 100

Building Data Pipeline Programming Language Kafka 100

Netflix Tech

NOVEMBER 14, 2023

Stateless Data Processing : As the name suggests, one should use this pattern in scenarios where the columns in the target table solely depend on the content of the incoming events, irrespective of their order of occurrence. A missed event in such a scenario would result in incorrect analysis due to a wrong derived state.

Data Process 93

Data Process Process Metadata Finance 93

Data Engineering Podcast

MARCH 22, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Metadata Hadoop Data Lake 100

Data Engineering Podcast

JULY 5, 2021

Summary At the core of every data pipeline is an workflow manager (or several). Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need.

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

FEBRUARY 20, 2022

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Confluent

MAY 21, 2019

This leads us to event streaming microservices patterns. Now that the profile change event is published, it can be received by the quote service. Now that the profile change event is published, it can be received by the quote service. In fact, schemas are more than just a contract between two event streaming microservices.

Kafka 110

Kafka Insurance Architecture Database 110

Data Engineering Podcast

MAY 22, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Machine Learning 130

Machine Learning Data Engineer Data Engineering Cloud 130

Data Engineering Podcast

NOVEMBER 20, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. What is the workflow for someone getting Sifflet integrated into their data stack?

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Ascend.io

NOVEMBER 28, 2023

The terms ‘data orchestration’ and ‘data pipeline orchestration’ are often used interchangeably, yet they diverge significantly in function and scope. Data orchestration refers to a wide collection of methods and tools that coordinate any and all types of data-related computing tasks.

Data Pipeline 52

Data Pipeline IT Data Data Engineer 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. RudderStack helps you build a customer data platform on your warehouse or data lake.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

JANUARY 23, 2022

Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. You can observe your pipelines with built in metadata search and column level lineage.

Management 100

Management Building Data Pipeline Metadata 100

Data Engineering Podcast

JULY 28, 2019

Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Data Integration 100

Data Integration Metadata Architecture Media 100

Cloudera

JUNE 16, 2022

CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report. The DevOps/app dev team wants to know how data flows between such entities and understand the key performance metrics (KPMs) of these entities.

Kafka 88

Kafka Manufacturing Data Lake SQL 88