Confluent

DECEMBER 8, 2021

Welcome Pythonistas to the streaming data world centered around Apache Kafka®! If you’re using Python and ready to get hands-on with Kafka, then you’re in the right place. This blog […].

Kafka 122

Kafka Python Data 122

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Kafka 145

Kafka Process Systems Engineering 145

Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 134

Kafka Data Ingestion Data Pipeline Cloud Storage 134

Confluent

MARCH 12, 2019

On the heels of part one in this blog series, Spring for Apache Kafka – Part 1: Error Handling, Message Conversion and Transaction Support , here in part two we’ll focus on another project that enhances the developer experience when building streaming applications on Kafka: Spring Cloud Stream. What is Spring Cloud Stream?

Kafka 111

Kafka Cloud Programming Coding 111

Confluent

JUNE 25, 2019

It’s official: Apache Kafka ® 2.3 Core Kafka. In order to keep your data safe, Kafka creates several replicas of it on different brokers. Kafka will not allow writes to proceed unless the partition has a minimum number of in-sync replicas. Kafka Connect. has been released! This is called the “minimum ISR.”.

Kafka 111

Kafka Accessible Accessibility IT 111

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on.

Kafka 55

Kafka Process Algorithm Cloud 55

Confluent

JULY 24, 2019

Using Jaeger tracing, I’ve been able to answer an important question that nearly every Apache Kafka ® project that I’ve worked on posed: how is data flowing through my distributed system? Distributed tracing with Apache Kafka and Jaeger. Example of a Kafka project with Jaeger tracing. What does this all mean?

Kafka 54

Kafka Systems Bytes Project 54

Cloudera

SEPTEMBER 20, 2021

There are many ways that Apache Kafka has been deployed in the field. In our Kafka Summit 2021 presentation, we took a brief overview of many different configurations that have been observed to date. In this blog series, we will discuss each of these deployments and the deployment choices made along with how they impact reliability.

Kafka 116

Kafka Systems Utilities Bytes 116

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. So why is Kafka such a good fit? Apache Kafka and domain-driven microservices.

Kafka 109

Kafka Designing Architecture Coding 109

DoorDash Engineering

MARCH 27, 2024

At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka. In Kafka, a test tenant processing production event can cause data inconsistencies, including outages and other incidents.

Kafka 103

Kafka Architecture Algorithm Process 103

Confluent

JULY 1, 2019

Put another way, courtesy of Spencer Ruport: LISTENERS are what interfaces Kafka binds to. Apache Kafka ® is a distributed system. You need to tell Kafka how the brokers can reach each other but also make sure that external clients (producers/consumers) can reach the broker they need to reach. Is anyone listening? on AWS, etc.)

Kafka 101

Kafka Metadata AWS Bytes 101

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This article describes how to instrument Kafka-based applications with distributed tracing capabilities in order to make dataflows between event-based components more visible.

Kafka 111

Kafka Transportation Metadata Consulting 111

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Database Medical 21

Confluent

SEPTEMBER 13, 2019

Static Membership is an enhancement to the current rebalance protocol that aims to reduce the downtime caused by excessive and unnecessary rebalances for general Apache Kafka ® client implementations. This applies to Kafka consumers, Kafka Connect, and Kafka Streams. What does “rebalance” mean when it comes to Kafka?

Kafka 21

Kafka Cloud Metadata Process 21

Confluent

NOVEMBER 13, 2020

Apache Kafka®. This is the final blog […]. Asynchronous boundaries. Frameworks. Configuring frameworks. All of these share one thing in common: complexity in testing. Now imagine them combined—it gets much harder.

Kafka 98

Kafka IT 98

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. However, Apache Kafka is more than just messaging. Some Kafka and Rockset users have also built real-time e-commerce applications , for example, using Rockset’s Java, Node.js

Kafka 21

Kafka SQL BI Hadoop 21

Confluent

AUGUST 1, 2019

The Kafka Streams API boasts a number of capabilities that make it well suited for maintaining the global state of a distributed system. At Imperva, we took advantage of Kafka Streams to build shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in our system.

Kafka 20

Kafka Systems Building Metadata 20

Confluent

JULY 16, 2019

How cool would it be to build your own burglar alarm system that can alert you before the actual event takes place simply by using a few network-connected cameras and analyzing the camera images with Apache Kafka ® , Kafka Streams, and TensorFlow? I will show how to implement this use case in this blog post.

Machine Learning 101

Machine Learning Kafka Java Datasets 101

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Microservices, Apache Kafka, and Domain-Driven Design (DDD) covers this in more detail. Example: Severstal.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. Calls the data plane Apache Kafka® Connect API to obtain information about the current state of the system, such as the status of currently running connectors and their configurations.

Kafka 50

Kafka MySQL Database Software Engineer 50

DoorDash Engineering

AUGUST 13, 2024

This journey began with Kafka, one of our most critical and widely used infrastructure components. Kafka is a distributed event streaming platform that DoorDash uses to handle billions of real-time events. To address this, we developed Kafka Self-Serve, our flagship self-serve storage infrastructure platform.

Kafka 82

Kafka Engineering AWS Designing 82

Confluent

JANUARY 29, 2021

This blog post shares SPOUD’s story of engineering a simple […]. What happens when you let engineers go wild building an application to track the score of a table football game?

Kafka 118

Kafka Engineering Building 118

Jesse Anderson

OCTOBER 5, 2023

A few examples: Everything should use Kafka – No, it shouldn’t. Kafka isn’t for everyone and everything. Kafka is easy – It’s not that simple. Confluent Cloud makes Kafka operations easier but doesn’t change the difficulty for architecture or development. The talks went from ksqlDB and Kafka Streams to Flink.

Kafka 195

Kafka Finance Cloud Designing 195

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. For more information regarding this, refer to our previous blog.

Datasets 100

Datasets Computer Science Systems Kafka 100

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Workfall

JUNE 27, 2023

It means that there is a high risk of data loss but Apache Kafka solves this because it is distributed and can easily scale horizontally and other servers can take over the workload seamlessly. This is where Apache Kafka comes in. Kafka can also be used to stream data from IoT devices or sensors. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Cloudera

SEPTEMBER 26, 2023

The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog. Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

Kafka 95

Kafka Technology IT Government 95

Netflix Tech

FEBRUARY 14, 2025

In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. These extracted events are then routed to an Apache Kafka topic for immediate processing needs and simultaneously stored in an Apache Iceberg table for long-term retention and historical analysis.

Kafka 66

Kafka Datasets Metadata Utilities 66

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 80

Kafka Data Cloud Big Data 80

Uber Engineering

OCTOBER 21, 2021

Apache Kafka ® is widely used across Uber’s multiple business lines. Take the example of an Uber ride: When a user opens up the Uber app, demand and supply data are aggregated in Kafka queues to serve fare calculations. Background.

Management 137

Management Kafka Engineering Data 137

Confluent

MAY 22, 2024

Scaling Kafka clusters up and down to meet changing demands is challenging. In this blog, learn how elastic autoscaling Enterprise clusters save you time and money.

Kafka 69

Kafka 69

Jesse Anderson

JUNE 12, 2024

We also discuss the various systems using Kafka’s protocol. Confluent has never shied away from saying Kafka is “easy,” and I disagree. During the Kafka Summit London Keynote, the speakers said “easy” 17 times; in the Kafka Summit Bangalore Keynote, they said it 18 times. Want to use the Kafka protocol with Pulsar?

Kafka 147

Kafka Data Lake Architecture Cloud 147

Data Engineering Weekly

FEBRUARY 23, 2025

I found the blog to be a comprehensive roadmap for data engineering in 2025. The blog narrates the gateway's purpose: simplifying access, enabling experimentation, achieving cost-efficiency, and providing auditing and platformization benefits.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Confluent

OCTOBER 25, 2023

Debugging Apache Kafka® issues shouldn’t send shivers down your spine. Explore the latest blog posts, on-demand videos, and demos on Kafka troubleshooting to ease your fears.

Kafka 78

Kafka 78

databricks

AUGUST 12, 2024

The blog explores data streams from NASA satellites using Apache Kafka and Databricks. It demonstrates ingestion and transformation with Delta Live Tables in SQL and AI/BI-powered analysis of supernova events.

BI 116

BI Kafka SQL Data 116