DoorDash Engineering

MARCH 27, 2024

At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka. In Kafka, a test tenant processing production event can cause data inconsistencies, including outages and other incidents.

Kafka 103

Kafka Architecture Algorithm Systems 103

Waitingforcode

MAY 4, 2023

Today I'll focus on these differences for Amazon Kinesis service and Apache Kafka ecosystem. The managed cloud services often share the same fundamentals as their Open alternatives. However, there is always something different.

Kafka 147

Kafka Cloud Management 147

DoorDash Engineering

AUGUST 13, 2024

This journey began with Kafka, one of our most critical and widely used infrastructure components. Kafka is a distributed event streaming platform that DoorDash uses to handle billions of real-time events. To address this, we developed Kafka Self-Serve, our flagship self-serve storage infrastructure platform.

Kafka 82

Kafka Engineering AWS Designing 82

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Confluent

SEPTEMBER 7, 2023

Kafka-on-Windows tutorials are everywhere, but most run Kafka directly on Windows. Here's how to use Kafka on Windows in a Linux environment backed by WSL 2, maximizing performance and stability

Kafka 116

Kafka 116

Confluent

SEPTEMBER 5, 2024

Dive into Kafka internals with a four-part series examining client requests and brokers. Part 1 covers what a producer does to prepare raw event data for the broker.

Kafka 92

Kafka Data 92

Confluent

OCTOBER 28, 2021

Kafka Streams is an abstraction over Apache Kafka® producers and consumers that lets you forget about low-level details and focus on processing your Kafka data. You could of course write […].

Kafka 131

Kafka Process Data 131

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Confluent

JULY 11, 2024

confluent-kafka-javascript offers two APIs for calling Kafka from JS, one KafkaJS-like & one node-rdkafka-like. Learn it by making an app w/producer, consumer & UI.

Kafka 78

Kafka Building IT 78

Confluent

MAY 17, 2022

I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. release contains many new features and improvements. This blog will highlight […].

Kafka 139

Kafka 139

Confluent

SEPTEMBER 16, 2021

The full inventory of three online Kafka Summits in 2021 is now complete. Kafka Summit Americas wrapped just yesterday. Being a part of the event team and the Program Committee, […].

Kafka 145

Kafka Programming 145

Waitingforcode

MAY 11, 2023

I have used to say "Kinesis Data Streams is like Apache Kafka, an append-only streaming broker with partitions and offsets". Although often it's true, it's not that simple unfortunately.

Kafka 130

Kafka IT Data 130

Confluent

MAY 16, 2024

Learn how to contribute to open source Apache Kafka by writing Kafka Improvement Proposals (KIPs) that solve problems and add features! Read on for real examples.

Kafka 88

Kafka 88

Confluent

NOVEMBER 29, 2022

The mission-critical deployments, the robust feature set, the long history all say that Kafka is an Enterprise-capable product. Apache Kafka is going 1.0!

Kafka 105

Kafka 105

Confluent

OCTOBER 11, 2023

Apache Kafka 3.6 brings Tiered Storage Early Access, migrating clusters from ZooKeeper to KRaft with no downtime, a grace period for stream-table joins, and more!

Kafka 90

Kafka Accessible Accessibility 90

Confluent

OCTOBER 3, 2022

Apache Kafka 3.3 includes KRaft mode, improves partition scalability and resiliency while simplifying Kafka deployment, as well as updates to Kafka Streams, Connect, and more.

Kafka 116

Kafka 116

Confluent

FEBRUARY 7, 2023

Migrate Kafka clusters from ZooKeeper to KRaft with no downtime (early access), get improvements for Kafka Streams and Kafka Connect, and more.

Kafka 105

Kafka Accessible Accessibility 105

Hevo

DECEMBER 26, 2023

Have you considered the simplest way to write and read messages from Kafka? If not, in this article you will discover how you can use Kafka Console Producer. You can use the Kafka Console Producer to write records to a Kafka Topic straight from the command line.

Kafka 52

Kafka Data 52

Confluent

MAY 9, 2023

Low utilization and operational complexity dramatically increases Kafka costs, so we reinvented Kafka as a cloud-native and complete service to reduce costs for thousands of businesses at any scale.

Kafka 107

Kafka Utilities Cloud 107

Uber Engineering

JULY 6, 2024

Kafka Tiered Storage, developed in collaboration with the Apache Kafka community, introduces the separation of storage and processing in brokers, significantly improving the scalability, reliability, and efficiency of Kafka clusters.

Kafka 67

Kafka Process Data 67

Confluent

SEPTEMBER 21, 2021

I’m pleased to announce the release of Apache Kafka 3.0 on behalf of the Apache Kafka® community. Apache Kafka 3.0 is a major release in more ways than one. Apache […].

Kafka 145

Kafka 145

Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 125

Kafka Data Ingestion Data Pipeline Cloud Storage 125

Confluent

MAY 15, 2023

Take a tour of the internals of Confluent’s Apache Kafka® service, powered by Kora: the next-generation, cloud-native streaming engine.Kora.

Kafka 145

Kafka Cloud Engineering 145

Confluent

MARCH 30, 2021

At the heart of Apache Kafka® sits the log—a simple data structure that uses sequential operations that work symbiotically with the underlying hardware. Efficient disk buffering and CPU cache usage, […].

Kafka 145

Kafka Data 145

Confluent

FEBRUARY 15, 2023

There are two main consumer group memberships in Apache Kafka®. Here’s how static and dynamic consumer groups work, how they affect rebalancing, and which to choose for your application.

Kafka 122

Kafka 122

Confluent

DECEMBER 2, 2022

To prioritize the safety of our community, we are transforming Kafka Summit Austin into a virtual experience. We are excited to invite the global Kafka community to Kafka Summit 2020: Event Streaming Everywhere.

Kafka 111

Kafka 111

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. On the other hand, log-based message brokers such as Apache Kafka combine the durability of database storage with the low-latency notification capabilities of messaging systems.

Kafka 77

Kafka Python Process Google Cloud 77

Confluent

DECEMBER 3, 2021

Each one of the more than 50 tutorials for Apache Kafka® on Confluent Developer answers a question that you might ask a knowledgeable friend or colleague about Kafka and its […].

Kafka 135

Kafka IT 135

Confluent

MAY 17, 2021

And that’s a wrap on Kafka Summit Europe 2021, the first of three global Kafka Summits this year. We’ve seen 17,000 registrations from over 7,000 companies and 137 different countries. […].

Kafka 123

Kafka 123

Confluent

SEPTEMBER 18, 2023

Learn how Apache Kafka message compression works, why and how to use it, the five types of compression, configurations for the compression type, and how messages are decompressed.

Kafka 72

Kafka IT 72

Confluent

SEPTEMBER 29, 2021

Apache Kafka® is an enormously successful piece of data infrastructure, functioning as the ubiquitous distributed log underlying the modern enterprise. It is scalable, available as a managed service, and has […].

Kafka 98

Kafka Management IT Data 98

Confluent

JUNE 4, 2024

Make a streaming stock chart with Alpaca (data), Kafka (orchestration), Flink SQL (processing), and Streamlit (UI/app). 1 (of 2) gets data via Kafka into Flink.

Kafka 75

Kafka SQL Process Data 75

Confluent

JANUARY 6, 2023

Introducing fully managed Apache Kafka® + Flink for the most robust, cloud-native data streaming platform with stream processing, integration, and streaming analytics in one.

Kafka 145

Kafka Cloud Process Management 145

Confluent

JANUARY 20, 2021

Apache Kafka® is at the core of a large ecosystem that includes powerful components, such as Kafka Connect and Kafka Streams. This ecosystem also includes many tools and utilities that […].

Kafka 130

Kafka Utilities 130

Confluent

MAY 11, 2023

Running Kafka is costly, but Confluent has created a far more efficient product to lower your costs. Join the Cost Savings challenge to see for yourself.

Kafka 142

Kafka 142

Confluent

MAY 22, 2024

Scaling Kafka clusters up and down to meet changing demands is challenging. In this blog, learn how elastic autoscaling Enterprise clusters save you time and money.

Kafka 69

Kafka 69

Confluent

JUNE 28, 2023

Get an introduction into the world of events and event-driven architecture in Apache Kafka. Learn what events are and the role they play in event design, event streaming, and event-driven design.

Kafka 111

Kafka Architecture Designing 111