Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems. ETL workflows), as well as downstream (e.g.

Systems 95

Systems Media Machine Learning Data Warehouse 95

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 89

Systems Metadata Media Kafka 89

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. We used Elasticsearch dashboards to analyze results.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Data Engineering Podcast

JULY 5, 2021

SaaSGlue is a managed service that lets you connect all of your systems, across clouds and physical infrastructure, and spanning all of your programming languages. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Atlan is the metadata hub for your data ecosystem. What are some of the unique characteristics of that information?

Building 130

Building IT Metadata MongoDB 130

Netflix Tech

NOVEMBER 14, 2023

Late-arriving data is essentially delayed data due to system retries, network delays, batch processing schedules, system outages, delayed upstream workflows, or reconciliation in source systems. Using fixed lookback windows to always reprocess data, assuming that most late-arriving events will occur within that window.

Data Engineering 109

Data Engineering Data Engineer Engineering Metadata 109

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. Yeah…I can do that already!

Process 83

Process Kafka SQL Datasets 83

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. Imagine a library with millions of books but no catalog system to organize them. This is what managing data without metadata feels like. What is Metadata? Chaos, right?

Metadata 52

Metadata IT Government High Quality Data 52

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously. Let’s imagine a “Hello, World!”

Kafka 111

Kafka Transportation Metadata Consulting 111

Data Engineering Podcast

JUNE 17, 2023

In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Data Engineering Podcast

MARCH 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. what are the characteristics that distinguish a knowledge graph from What are the layers/stages of applications and data that can/should incorporate JSON-LD as the representation for records and events?

Building 189

Building SQL BI Python 189

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. This rule empowers microservices to change their database schemas however they like without worrying about breaking other systems. What are Events?

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

MARCH 18, 2023

He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Cloudera

NOVEMBER 13, 2020

The commonly-accepted best practice in database system design for years is to use an exhaustive search strategy to consider all the possible variations of specific database operations in a query plan. Metadata Caching. See the performance results below for an example of how metadata caching helps reduce latency.

Metadata 144

Metadata Coding SQL Database 144

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. An OS snapshot is a collection of bootfiles (initrd, vmlinuz), RPMs, and a few extra metadata.

Systems 55

Systems Metadata Datasets Designing 55

Maxime Beauchemin

JANUARY 20, 2017

This discipline also integrates specialization around the operation of so called “big data” distributed systems, along with concepts around the extended Hadoop ecosystem, stream processing, and in computation at scale. Those systems have been taught to normalize the data for storage on their own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Netflix Tech

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases.

Bytes 96

Bytes Datasets Metadata Data 96

Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. Let’s first examine what we would want from a typical system that processes a large amount of data for a critical cause.

Kafka 111

Kafka Accessible Accessibility Metadata 111

Netflix Tech

DECEMBER 14, 2018

In this post we will provide details of the NMDB system architecture beginning with the system requirements?—?these A fundamental requirement for any lasting data system is that it should scale along with the growth of the business applications it wishes to serve. key value stores generally allow storing any data under a key).

Media 97

Media Database Metadata Data Schemas 97

Maxime Beauchemin

JANUARY 7, 2018

When functions are “pure” — meaning they do not have side-effects — they can be written, tested, reasoned-about and debugged in isolation, without the need to understand external context or history of events surrounding its execution. Knowing when events were reported in relation to when they occurred is useful.

Data Engineering 183

Data Engineering Data Engineer Data Process Process 183

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 102

Metadata Data Pipeline Scala Data Workflow 102

Cloudera

AUGUST 10, 2021

Apache Ozone enhancements deliver full High Availability providing customers with enterprise-grade object storage and compatibility with Hadoop Compatible File System and S3 API. . Additional databases, operating systems, and development environments are now supported for better integration and compatibility , including support for RHEL8. .

Cloud 98

Cloud Kafka Metadata SQL 98

Confluent

APRIL 11, 2019

We knew that our users needed three things from the connector: Integration with the Kafka Connect API: Connect’s scaling and fault tolerance capabilities were important to have, and users didn’t want yet another system that they’d need to learn how to use, deploy and monitor. And no one likes missing events.

Kafka 110

Kafka AWS Metadata Architecture 110

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. The system can fail or apply a circuit breaker in the pipeline if the data contract fails. Event Routers typically don’t alter the payload.

Engineering 112

Engineering Kafka Data Pipeline Data Warehouse 112

Scott Logic

APRIL 22, 2024

I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment. I’ll use the term message in this blog as a generic term for asynchronous communication that may be state or commands or events.

AWS 52

AWS Metadata Systems Kafka 52

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Netflix Tech

SEPTEMBER 2, 2020

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. The more complex a system, the more places to look for clues. In an earlier blog post, we discussed Telltale , our health monitoring system. What is Edgar?

Metadata 119

Metadata Transportation Engineering Systems 119

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems. What are the alternatives to CDC?

Data Warehouse 100

Data Warehouse Metadata Hadoop Data Lake 100

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Netflix Tech

SEPTEMBER 18, 2024

HashMap<String, SortedMap<Bytes, Bytes>> For complex data models such as structured Records or time-ordered Events, this two-level approach handles hierarchical structures effectively, allowing related data to be retrieved together. This model supports both simple and complex data models, balancing flexibility and efficiency.

Bytes 101

Bytes Metadata Database Data 101

Engineering at Meta

FEBRUARY 6, 2024

DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote artifact for the current operating system and CPU. Our continuous integration (CI) system supports special configuration for DotSlash jobs where a user must specify: A set of builds to run (these can span multiple platforms).

Metadata 133

Metadata Coding Building Project 133

Data Engineering Podcast

FEBRUARY 27, 2022

Time-series data is time stamped so you can measure how a system is changing. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. What other tools/systems will need to integrate with it?

Designing 100

Designing Metadata Data Lake Relational Database 100