Confluent

JULY 8, 2020

In the article Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann discusses when to combine several event types in the same topic and introduces new […].

Kafka 138

Kafka 138

Confluent

FEBRUARY 2, 2021

Since I first started using Apache Kafka® eight years ago, I went from being a student who had just heard about event streaming to contributing to the transformational, company-wide event […].

Kafka 97

Kafka Data Schemas Cloud Data 97

Confluent

SEPTEMBER 21, 2022

A deep dive into how microservices work, why it’s the backbone of real-time applications, and how to build event-driven microservices applications with Python and Kafka.

Kafka 98

Kafka Python Building 98

Data Engineering Podcast

APRIL 23, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Confluent

OCTOBER 16, 2020

Apache Kafka® is an event streaming platform used by more than 30% of the Fortune 500 today. There are numerous features of Kafka that make it the de-facto standard for […].

Kafka 145

Kafka IT 145

Confluent

SEPTEMBER 5, 2024

Dive into Kafka internals with a four-part series examining client requests and brokers. Part 1 covers what a producer does to prepare raw event data for the broker.

Kafka 139

Kafka Data 139

Confluent

SEPTEMBER 24, 2020

Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring […].

Kafka 143

Kafka Engineering Systems Process 143

Confluent

AUGUST 21, 2020

Apache Kafka® is one of the most popular event streaming systems. Kafka […]. There are many ways to compare systems in this space, but one thing everyone cares about is performance.

Kafka 145

Kafka Systems 145

Confluent

AUGUST 8, 2019

We’re excited to announce Tutorials for Apache Kafka ® , a new area of our website for learning event streaming. Kafka Tutorials is a collection of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions. Lastly, Kafka Tutorials is a community-driven site.

Kafka 22

Kafka Data Warehouse Programming Coding 22

Confluent

AUGUST 20, 2019

This article presents an event-based architecture that retains most transactional properties as provided by an RDBMS, while leveraging Apache Kafka ® as a scalable and highly available single source of truth. Upholding each property in a system based on Kafka is tricky but not impossible, as you are about to find out.

Kafka 22

Kafka Systems Building Relational Database 22

Confluent

NOVEMBER 30, 2022

Putting Apache Kafka To Use: A Practical Guide to Building an Event Streaming Platform.

Kafka 105

Kafka Building Process 105

Confluent

MARCH 11, 2020

In the Apache Kafka® ecosystem, ksqlDB and Kafka Streams are two popular tools for building event streaming applications that are tightly integrated with Apache Kafka. While ksqlDB and Kafka Streams […].

Kafka 115

Kafka Process Building 115

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. Store streams of events in a fault-tolerant way. Microservices.

Kafka 109

Kafka Designing Architecture Coding 109

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on. Here we go!

Kafka 55

Kafka Process Algorithm Cloud 55

Confluent

NOVEMBER 25, 2020

As part of this year’s Kafka Summit, the Confluent Community team hosted a community hackathon named Kafkathon 2020. The event provided an opportunity for participants to learn, build, and showcase […].

Kafka 105

Kafka Building 105

Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 137

Kafka Data Ingestion Data Pipeline Cloud Storage 137

DoorDash Engineering

MARCH 27, 2024

Real-time event processing is a critical component of a distributed system’s scalability. At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka.

Kafka 103

Kafka Architecture Algorithm Process 103

Confluent

DECEMBER 8, 2020

If you’re getting started with Apache Kafka® and event streaming applications, you’ll be pleased to see the variety of languages available to start interacting with the event streaming platform. It […].

Kafka 122

Kafka Scala IT 122

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 101

Datasets Computer Science Systems Kafka 101

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Confluent

DECEMBER 2, 2022

To prioritize the safety of our community, we are transforming Kafka Summit Austin into a virtual experience. We are excited to invite the global Kafka community to Kafka Summit 2020: Event Streaming Everywhere.

Kafka 111

Kafka 111

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

JANUARY 7, 2021

At Zendesk, Apache Kafka® is one of our foundational services for distributing events among different internal systems. We have pods, which can be thought of as isolated cloud environments where […].

Kafka 119

Kafka Cloud Systems 119

Workfall

JUNE 27, 2023

It means that there is a high risk of data loss but Apache Kafka solves this because it is distributed and can easily scale horizontally and other servers can take over the workload seamlessly. This is where Apache Kafka comes in. Kafka can also be used to stream data from IoT devices or sensors. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Confluent

MARCH 29, 2021

Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and […].

Kafka 131

Kafka Systems Management IT 131

Confluent

JUNE 25, 2019

It’s official: Apache Kafka ® 2.3 Core Kafka. In order to keep your data safe, Kafka creates several replicas of it on different brokers. Kafka will not allow writes to proceed unless the partition has a minimum number of in-sync replicas. Kafka Connect. has been released! This is called the “minimum ISR.”.

Kafka 111

Kafka Accessibility Accessible IT 111

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Database Medical 21

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 119

Kafka Systems Architecture Building 119

Cloudera

SEPTEMBER 20, 2021

There are many ways that Apache Kafka has been deployed in the field. In our Kafka Summit 2021 presentation, we took a brief overview of many different configurations that have been observed to date. Kafka as software falls more cleanly into the Parallel Systems Reliability discussed below but some parts of it can end up Serial.

Kafka 116

Kafka Systems Utilities Bytes 116

Confluent

MAY 28, 2020

For many organizations, Apache Kafka® is the backbone and source of truth for data systems across the enterprise. Protecting your event streaming platform is critical for data security and often […].

Kafka 111

Kafka Data Security Systems Data 111

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. An event is a small, self-contained object that contains the details of something happened at some point in time e.g. user interaction. An event is generated by a producer (e.g.

Kafka 98

Kafka Python Process Google Cloud 98

Netflix Tech

FEBRUARY 14, 2025

Collecting Raw Impression Events As Netflix members explore our platform, their interactions with the user interface spark a vast array of raw events. These events are promptly relayed from the client side to our servers, entering a centralized event processing queue.

Kafka 66

Kafka Datasets Metadata Utilities 66

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

JANUARY 16, 2020

Now that we’ve learned about the processing layer of Apache Kafka® by looking at streams and tables, as well as the architecture of distributed processing with the Kafka Streams API […].

Kafka 26

Kafka Architecture Process 26

Confluent

JUNE 26, 2020

We are excited to announce the preview release of the fully managed Snowflake sink connector in Confluent Cloud, our fully managed event streaming service based on Apache Kafka®. Our managed […].

Kafka 105

Kafka Cloud Management 105

Confluent

MARCH 5, 2021

Most companies who have adopted event streaming are running multiple Apache Kafka® environments. For example, they may use different Kafka clusters for testing vs. production or for different use cases. […].

Kafka 98

Kafka Management 98