Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Kafka 135

Kafka Process Systems Engineering 135

Confluent

FEBRUARY 2, 2021

Since I first started using Apache Kafka® eight years ago, I went from being a student who had just heard about event streaming to contributing to the transformational, company-wide event […].

Kafka 97

Kafka Data Schemas Cloud Data 97

Confluent

SEPTEMBER 5, 2024

Dive into Kafka internals with a four-part series examining client requests and brokers. Part 1 covers what a producer does to prepare raw event data for the broker.

Kafka 92

Kafka Data 92

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Confluent

SEPTEMBER 16, 2021

The full inventory of three online Kafka Summits in 2021 is now complete. Kafka Summit Americas wrapped just yesterday. Being a part of the event team and the Program Committee, […].

Kafka 145

Kafka Programming 145

DoorDash Engineering

AUGUST 13, 2024

This journey began with Kafka, one of our most critical and widely used infrastructure components. Kafka is a distributed event streaming platform that DoorDash uses to handle billions of real-time events. To address this, we developed Kafka Self-Serve, our flagship self-serve storage infrastructure platform.

Kafka 82

Kafka Engineering AWS Designing 82

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. An event is a small, self-contained object that contains the details of something happened at some point in time e.g. user interaction. An event is generated by a producer (e.g.

Kafka 76

Kafka Python Process Google Cloud 76

Confluent

DECEMBER 2, 2022

To prioritize the safety of our community, we are transforming Kafka Summit Austin into a virtual experience. We are excited to invite the global Kafka community to Kafka Summit 2020: Event Streaming Everywhere.

Kafka 111

Kafka 111

Confluent

NOVEMBER 30, 2022

Putting Apache Kafka To Use: A Practical Guide to Building an Event Streaming Platform.

Kafka 105

Kafka Building Process 105

Confluent

APRIL 11, 2022

In just a few weeks’ time, the Apache Kafka® community will be convening for Kafka Summit London 2022—its first in-person event in over two years. The conference is being held […].

Kafka 52

Kafka IT 52

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Confluent

APRIL 24, 2023

Kafka Summit 2023 brings 60+ sessions, keynotes, and lightning talks, and more from industry leaders. Check out the agenda, highlights, networking events, and more event info.

Kafka 75

Kafka 75

Confluent

MAY 13, 2021

This is made possible by event-driven applications leveraging Apache Kafka® and built with fault tolerance […]. Failures are inevitable in any system, and there are various options for mitigating them automatically.

Kafka 95

Kafka Systems 95

Confluent

JUNE 11, 2019

For event streaming application developers, it is important to continuously update the streaming pipeline based on the need for changes in the individual applications in the pipeline. It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline.

Kafka 86

Kafka Cloud Java MongoDB 86

Confluent

JULY 8, 2020

In the article Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann discusses when to combine several event types in the same topic and introduces new […].

Kafka 138

Kafka 138

Hevo

SEPTEMBER 13, 2024

As the dependency on high-quality, real-time data availability increases, the need for event/data streaming tools becomes increasingly crucial. Today, we will explore the Top 5 […]

Kafka 52

Kafka Data Engineering Data Engineer Engineering 52

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 126

Kafka Data Ingestion Data Pipeline Cloud Storage 126

Confluent

NOVEMBER 25, 2020

As part of this year’s Kafka Summit, the Confluent Community team hosted a community hackathon named Kafkathon 2020. The event provided an opportunity for participants to learn, build, and showcase […].

Kafka 105

Kafka Building 105

Confluent

APRIL 17, 2024

Kafka Summit Bangalore 2024 brings 35+ sessions, keynotes, lightning talks, and more from industry leaders. Check out the agenda, highlights, networking events, and more event info.

Kafka 69

Kafka Data 69

Confluent

OCTOBER 16, 2020

Apache Kafka® is an event streaming platform used by more than 30% of the Fortune 500 today. There are numerous features of Kafka that make it the de-facto standard for […].

Kafka 145

Kafka IT 145

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 119

Kafka Systems Architecture Building 119

Confluent

MARCH 26, 2024

In this guide, you will learn how to track events in a large codebase like GitHub using Apache Kafka and Kafka Streams.

Kafka 69

Kafka Process Data 69

Confluent

FEBRUARY 28, 2024

Kafka Summit London 2024 brings 90+ sessions, keynotes, lightning talks, and more from industry leaders. Check out the agenda, highlights, networking events, and more event info.

Kafka 69

Kafka 69

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Confluent

APRIL 20, 2020

The goal of this post is to illustrate PUSH to web from Apache Kafka® with a hands-on example. Our business users are always wanting their data faster so they can […].

Kafka 94

Kafka Java Data 94

Rock the JVM

JANUARY 21, 2023

Giannis is a proud alumnus of Rock the JVM, working as a Solutions Architect with a focus on Event Streaming and Stream Processing Systems. Introduction Apache Kafka is a well-known event streaming platform used in many organizations worldwide. Environment Setup First, we want to have a Kafka Cluster up and running.

Kafka 69

Kafka Java Scala Coding 69

Confluent

SEPTEMBER 24, 2020

Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring […].

Kafka 143

Kafka Engineering Systems Process 143

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. Event-first FaaS? Event-first FaaS.

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

NOVEMBER 30, 2022

This is the second part of our guide on streaming data and Apache Kafka. This guide will contain specific advice on how to go about building an event streaming platform in your organization.

Kafka 52

Kafka Building Data Process 52

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. Four pillars of event streaming. Pillar 4 – Operational plane: Event logging, DLQs and automation. To read the other articles in this series, see: Journey to Event Driven – Part 1: Why Event-First Thinking Changes Everything.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Confluent

MARCH 11, 2020

In the Apache Kafka® ecosystem, ksqlDB and Kafka Streams are two popular tools for building event streaming applications that are tightly integrated with Apache Kafka. While ksqlDB and Kafka Streams […].

Kafka 115

Kafka Process Building 115

Confluent

AUGUST 21, 2020

Apache Kafka® is one of the most popular event streaming systems. Kafka […]. There are many ways to compare systems in this space, but one thing everyone cares about is performance.

Kafka 145

Kafka Systems 145

Confluent

APRIL 22, 2021

Why should you monitor your Apache Kafka® client applications? Apart from the usual reasons for monitoring any application, such as ensuring uptime SLAs, there are a few specific reasons for […].

Kafka 72

Kafka Cloud 72

Rockset

MARCH 5, 2020

Apache Kafka has seen broad adoption as the streaming platform of choice for building applications that react to streams of data in real time. In many organizations, Kafka is the foundational platform for real-time event analytics, acting as a central location for collecting event data and making it available in real time.

Kafka 40

Kafka Data Warehouse Data Lake Relational Database 40