Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. Calls the data plane Apache Kafka® Connect API to obtain information about the current state of the system, such as the status of currently running connectors and their configurations.

Kafka 50

Kafka MySQL Database Software Engineer 50

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

Data Engineering Weekly

APRIL 20, 2025

The blog took out the last edition’s recommendation on AI and summarized the current state of AI adoption in enterprises. The simplistic model expressed in the blog made it easy for me to reason about the transactional system design. Kafka is probably the most reliable data infrastructure in the modern data era.

Data Engineer 68

Data Engineer Data Engineering Engineering Kafka 68

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Confluent

APRIL 10, 2019

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 87

Kafka Java Metadata Cloud 87

Confluent

APRIL 30, 2019

With the release of Apache Kafka ® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. In what follows, we provide some context around how a processor topology was generated inside Kafka Streams before 2.1, Kafka Streams topology generation 101.

Kafka 91

Kafka Coding Process Software Engineering 91

Netflix Tech

FEBRUARY 14, 2025

In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. These extracted events are then routed to an Apache Kafka topic for immediate processing needs and simultaneously stored in an Apache Iceberg table for long-term retention and historical analysis.

Kafka 66

Kafka Datasets Utilities Metadata 66

Data Engineering Weekly

FEBRUARY 23, 2025

It covers nine categories: storage systems, data lake platforms, processing, integration, orchestration, infrastructure, ML/AI, metadata management, and analytics. I found the blog to be a comprehensive roadmap for data engineering in 2025. The proposal discusses how Kafka will implement queue functionality similar to SQS and RabbitMQ.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Data Engineering Weekly

APRIL 6, 2025

The Grab blog delights me since I have tried to do this many times. A cross-encoder teacher model, fine-tuned on human-labeled data and enriched Pin metadata, was distilled into a lightweight student model using semi-supervised learning over billions of impressions. Kudos to the Grab team for building a docs-as-code system.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Data Engineering Weekly

JUNE 16, 2024

Get Your Copy → Walmart: Reliably Processing Trillions of Kafka Messages Per Day Consumer rebalancing and Head-of-line (HOL) blocking are some of the most common challenges while operating Kafka at scale. The blog explains KIP-932 and its potential benefits. The blog explains KIP-932 and its potential benefits.

Data Engineer 105

Data Engineer Data Engineering Engineering Kafka 105

Cloudera

AUGUST 10, 2021

We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Deep Dive 2: Atlas / Kafka integration. x, and 6.3.x,

Cloud 97

Cloud Kafka Metadata SQL 97

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Data and Metadata: Data inputs and data outputs produced based on the application logic. Introduction.

Architecture 124

Architecture Metadata Kafka Government 124

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Cloudera

SEPTEMBER 28, 2020

The customer also wanted to utilize the new features in CDP PvC Base like Apache Ranger for dynamic policies, Apache Atlas for lineage, comprehensive Kafka streaming services and Hive 3 features that are not available in legacy CDH versions. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams. Kafka, SRM, SMM.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. Databricks bought Tabular for $1b.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. Sample repository. gradlew composeUp.

Kafka 96

Kafka Management Bytes SQL 96

Christophe Blefari

JULY 13, 2024

It provides abstractions and tools for the translation of lakehouse table format metadata. You can also read Timo detailed post on Mixpanel blog and why SCD are the best thing for product analytics. Apache Kafka overview — If you're not familiar with Kafka this is a great overview. His chart are great.

Kafka 130

Kafka AWS Data Database 130

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. This time Apple engineers shared at Trino—good old Presto for lost people—Summit how they use it.

Kafka 130

Kafka BI Finance Data 130

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 111

Kafka Java Software Engineer Software Engineering 111

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Hevo]([link] Are you sick of repetitive, time-consuming ELT work?

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Cloudera

OCTOBER 25, 2021

From this release, Streams Messaging templates will support scaling with automatic rebalancing allowing you to grow or shrink your Apache Kafka cluster based on demand. Kafka Scaling. There will now be two specific hostgroups for Kafka Brokers; These are the Core_broker and Broker host groups. Atlas Schema Registry Integration.

Cloud 91

Cloud Kafka Metadata Management 91

Pinterest Engineering

NOVEMBER 22, 2023

In the first blog, we will share a short summary on the GokuS and GokuL architecture, data format for Goku Long Term, and how we improved the bootstrap time for our storage and serving components. This is because the synthetic data points would be present in the retry kafka waiting to be pushed into the recovering host by the retry ingestor.

Database 111

Database Bytes Kafka Architecture 111

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. Why is Data Quality Expensive? Now, Why is Data Quality Expensive?

Engineering 111

Engineering Kafka Data Pipeline Data Warehouse 111

Cloudera

MARCH 2, 2023

In this blog post, we are going to share with you how Cloudera Stream Processing ( CSP ) is integrated with Apache Iceberg and how you can use the SQL Stream Builder ( SSB ) interface in CSP to create stateful stream processing jobs using SQL. Iceberg is a high-performance open table format for huge analytic data sets.

Process 114

Process SQL Kafka Database 114

Netflix Tech

MARCH 10, 2023

We built an asset management platform (AMP), codenamed Amsterdam , in order to easily organize and manage the metadata, schema, relations and permissions of these assets. Amsterdam service utilizes various solutions such as Cassandra , Kafka , Zookeeper , EvCache etc. This is the layer we’d like to focus on in this blog.

Management 105

Management Metadata Digital Media Kafka 105

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Introduction and Rationale. Networking . Security integration.

Architecture 102

Architecture Cloud Kafka Hadoop 102

Cloudera

JUNE 16, 2022

In 2015, Cloudera became one of the first vendors to provide enterprise support for Apache Kafka, which marked the genesis of the Cloudera Stream Processing (CSP) offering. Today, CSP is powered by Apache Flink and Kafka and provides a complete, enterprise-grade stream management and stateful processing solution. Who is affected?

Kafka 88

Kafka Manufacturing Data Lake SQL 88

Confluent

MAY 21, 2019

The profile service will publish the changes in profiles, including address changes to an Apache Kafka ® topic, and the quote service will subscribe to the updates from the profile changes topic, calculate a new quote if needed and publish the new quota to a Kafka topic so other services can subscribe to the updated quote event.

Kafka 110

Kafka Insurance Architecture Database 110

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! At Netflix, our backend microservices continuously generate real-time event data that gets streamed into Kafka. Given our role on this critical path, accuracy is paramount.

Data Engineer 108

Data Engineer Data Engineering Engineering Metadata 108

Confluent

AUGUST 27, 2019

We are excited to announce the release of Confluent Cloud Schema Registry in general availability (GA), available in Confluent Cloud , our fully managed event streaming service based on Apache Kafka ®. Confluent Schema Registry provides a serving layer for your metadata and a RESTful interface for storing and retrieving Avro schemas.

Cloud 18

Cloud Kafka Electronics Metadata 18

Scribd Technology

MAY 18, 2021

Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd’s data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. To serve this need, we created kafka-delta-ingest.

Kafka 52

Kafka Data Warehouse Bytes Metadata 52

Cloudera

AUGUST 10, 2021

Since the purpose of this blog is to show lineage for data that exists in Ozone, I’m going to do a simple transformation in the Spark shell and write the data out to Ozone. If you want to continue experimenting with Ozone and Atlas, you can try writing to Ozone via Kafka using our documented configuration examples.

Hadoop 105

Hadoop Kafka Datasets Government 105

Netflix Tech

MARCH 25, 2019

In the rest of this blog, we will a) touch on the complexity of Netflix cloud landscape, b) discuss lineage design goals, ingestion architecture and the corresponding data model, c) share the challenges we faced and the learnings we picked up along the way, and d) close it out with “what’s next” on this journey. push or pull.

Building 90

Building Metadata Transportation Data Ingestion 90

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. The example 1_typedef-server.json describes the server typedef used in this blog. .

Data Governance 97

Data Governance Government Metadata Datasets 97

Pinterest Engineering

SEPTEMBER 17, 2024

Jeff Xiang | Senior Software Engineer, Logging Platform; Vahid Hashemian | Staff Software Engineer, LoggingPlatform When it comes to PubSub solutions, few have achieved higher degrees of ubiquity, community support, and adoption than Apache Kafka, which has become the industry standard for data transportation at large scale.

Kafka 49

Kafka Bytes Transportation Metadata 49

Rockset

MARCH 18, 2024

While it's well-known that Flink excels at filtering, joining and enriching streaming data from Apache Kafka® or Confluent Cloud , what is less known is that it is increasingly becoming ingrained in the end-to-end stack for AI-powered applications. These additional inputs are referred to as metadata filtering. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64