Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Netflix Tech

FEBRUARY 14, 2025

Collecting Raw Impression Events As Netflix members explore our platform, their interactions with the user interface spark a vast array of raw events. These events are promptly relayed from the client side to our servers, entering a centralized event processing queue.

Kafka 66

Kafka Datasets Metadata Utilities 66

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

MARCH 20, 2019

Back in May 2017, we laid out why we believe that Kafka Streams is better off without a concept of watermarks or triggers , and instead opts for a continuous refinement model. By continuous refinement , I mean that Kafka Streams emits new results whenever records are updated. Use case: Alerting. But how can you build alerts?

Kafka 106

Kafka Programming Process Designing 106

Snowflake

NOVEMBER 11, 2024

With Snowpipe for Apache Kafka (public preview soon in AWS and Microsoft Azure), a “pull” mechanism, rather than the existing “push” connector, allows you to extract and ingest Apache Kafka events into your Snowflake account directly without hosting your own Kafka Connect cluster.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. Useful to know in case of a disaster event). How far behind is data replication?

Kafka 87

Kafka Java Metadata Cloud 87

Pinterest Engineering

NOVEMBER 18, 2024

This configuration ensures that if the host goes down due to an EC2® event or any other reason, it will be automatically reprovisioned. Calls the data plane Apache Kafka® Connect API to obtain information about the current state of the system, such as the status of currently running connectors and their configurations.

Kafka 50

Kafka MySQL Database Software Engineer 50

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. FROM SYSMON_JOIN.

Process 83

Process Kafka SQL Datasets 83

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Data Engineering Weekly

FEBRUARY 23, 2025

Try Astro Free → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. The proposal discusses how Kafka will implement queue functionality similar to SQS and RabbitMQ.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Data Engineering Weekly

APRIL 6, 2025

You won't want to miss this live event on April 23rd! A cross-encoder teacher model, fine-tuned on human-labeled data and enriched Pin metadata, was distilled into a lightweight student model using semi-supervised learning over billions of impressions. Introducing Apache Airflow® 3.0 Be among the first to see Airflow 3.0

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. Introduction. gradlew composeUp.

Kafka 96

Kafka Management Bytes SQL 96

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. Databricks bought Tabular for $1b.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Cloudera

OCTOBER 7, 2021

Data and Metadata: Data inputs and data outputs produced based on the application logic. Also included, business and technical metadata, related to both data inputs / data outputs, that enable data discovery and achieving cross-organizational consensus on the definitions of data assets.

Architecture 124

Architecture Metadata Kafka Government 124

Confluent

APRIL 2, 2019

Includes free forever Confluent Platform on a single Apache Kafka ® broker, improved Control Center functionality at scale and hybrid cloud streaming. the event streaming platform built by the original creators of Apache Kafka. What do we mean by contextual event-driven applications? Confluent Platform 5.2

Kafka 21

Kafka Java Cloud Metadata 21

Cloudera

AUGUST 10, 2021

Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Deep Dive 2: Atlas / Kafka integration. This will expose newly created Kafka topics to Atlas.

Cloud 97

Cloud Kafka Metadata SQL 97

Data Engineering Weekly

MAY 16, 2023

Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. Like the staging environment, Fronting Kafka receives all the events without validation. Now, Why is Data Quality Expensive?

Engineering 111

Engineering Kafka Data Pipeline Data Warehouse 111

Confluent

MAY 21, 2019

This leads us to event streaming microservices patterns. Now that the profile change event is published, it can be received by the quote service. Now that the profile change event is published, it can be received by the quote service. In fact, schemas are more than just a contract between two event streaming microservices.

Kafka 110

Kafka Insurance Architecture Database 110

Rock the JVM

JANUARY 21, 2023

Giannis is a proud alumnus of Rock the JVM, working as a Solutions Architect with a focus on Event Streaming and Stream Processing Systems. Introduction Apache Kafka is a well-known event streaming platform used in many organizations worldwide. Environment Setup First, we want to have a Kafka Cluster up and running.

Kafka 69

Kafka Java Scala Coding 69

Netflix Tech

NOVEMBER 14, 2023

Using fixed lookback windows to always reprocess data, assuming that most late-arriving events will occur within that window. Add alerts to flag when late arriving data appears, block the pipelines, and perform a manual intervention where we triggered backfill pipelines to handle the missed events. Some techniques we used were: 1.

Data Engineer 108

Data Engineer Data Engineering Engineering Metadata 108

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

MAY 18, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. What is the current state of the ecosystem for generating and sharing metadata between systems? What are your goals for the OpenLineage effort?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Christophe Blefari

JULY 13, 2024

It provides abstractions and tools for the translation of lakehouse table format metadata. Apache Kafka overview — If you're not familiar with Kafka this is a great overview. Stories How Canva collects 25 billion events per day — Protobuf + Amazon Kinesis.

Kafka 130

Kafka AWS Data Database 130

Cloudera

OCTOBER 25, 2021

From this release, Streams Messaging templates will support scaling with automatic rebalancing allowing you to grow or shrink your Apache Kafka cluster based on demand. Kafka Scaling. There will now be two specific hostgroups for Kafka Brokers; These are the Core_broker and Broker host groups.

Cloud 91

Cloud Kafka Metadata Management 91

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Cloudera

JUNE 16, 2022

In 2015, Cloudera became one of the first vendors to provide enterprise support for Apache Kafka, which marked the genesis of the Cloudera Stream Processing (CSP) offering. Today, CSP is powered by Apache Flink and Kafka and provides a complete, enterprise-grade stream management and stateful processing solution. Who is affected?

Kafka 88

Kafka Manufacturing Data Lake SQL 88

Netflix Tech

JUNE 7, 2021

Flow Collector consumes two data streams, the IP address change events from Sonar via Kafka and eBPF flow log data from the Flow Exporter sidecars. It performs real time attribution of flow data with application metadata from Sonar. Sonar is an IPv6 and IPv4 address identity tracking service.

Transportation 131

Transportation AWS Cloud Kafka 131

Confluent

AUGUST 27, 2019

We are excited to announce the release of Confluent Cloud Schema Registry in general availability (GA), available in Confluent Cloud , our fully managed event streaming service based on Apache Kafka ®. Confluent Schema Registry provides a serving layer for your metadata and a RESTful interface for storing and retrieving Avro schemas.

Cloud 18

Cloud Kafka Electronics Metadata 18

Cloudera

MARCH 15, 2023

Introduction Many modern application designs are event-driven. An event-driven architecture enables minimal coupling, which makes it an optimal choice for modern, large-scale distributed systems. Send an event to notify other services about the new order. These services might be responsible for checking the inventory (eg.

PostgreSQL 91

PostgreSQL Kafka Database Data 91

Confluent

SEPTEMBER 16, 2019

Running a single Apache Kafka ® cluster across multiple datacenters (DCs) is a common, yet somewhat taboo architecture. This functionality allows operators to increase data durability and automate client failover in the event of a disaster. Essentially, this is how Apache Kafka provides durability. Follower Fetching.

Kafka 18

Kafka Metadata Architecture Software Engineer 18

Lyft Engineering

JUNE 28, 2023

The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. The interface was designed such that a minimal amount of metadata was needed to construct a pipeline object which performs a given capability. One key component is the Analytics Event Abstraction layer.

Machine Learning 120

Machine Learning Building Kafka Metadata 120

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Machine Learning 100

Christophe Blefari

DECEMBER 9, 2022

It happens for instance when you have 2 producers of the same event but they are not using the same type for a column. Using server sent events to simplify real-time streaming at scale — Interesting discussion about concepts around real-time communication for apps. in Terraform) but for data.

SQL 130

SQL Data Data Science Kafka 130

Afterpay Tech

SEPTEMBER 6, 2022

Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. Context The asynchronous processing capability that Kafka offers opens up numerous innovation opportunities to interact with other services.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Scribd Technology

MAY 18, 2021

Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd’s data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. To serve this need, we created kafka-delta-ingest.

Kafka 52

Kafka Data Warehouse Bytes Metadata 52