Confluent

AUGUST 1, 2019

The Kafka Streams API boasts a number of capabilities that make it well suited for maintaining the global state of a distributed system. At Imperva, we took advantage of Kafka Streams to build shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in our system.

Kafka 20

Kafka Systems Building Metadata 20

Confluent

SEPTEMBER 6, 2019

First, what is event sourcing? We can answer all those questions because the individual events that make up our balance are stored. In fact, it’s the summation of these events that result in our current account balance. This, in a nutshell, is event sourcing. Event sourcing: primary vs. derivative.

Kafka 22

Kafka MySQL Banking Java 22

Confluent

SEPTEMBER 24, 2020

Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring […].

Kafka 143

Kafka Engineering Systems Process 143

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Use cases for IoT technologies and an event streaming platform. Example: E.ON. Example: Target. Example: Severstal.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. Event-first FaaS? Event-first FaaS.

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Confluent

AUGUST 8, 2019

We’re excited to announce Tutorials for Apache Kafka ® , a new area of our website for learning event streaming. Kafka Tutorials is a collection of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions. Lastly, Kafka Tutorials is a community-driven site.

Kafka 22

Kafka Data Warehouse Programming Coding 22

DoorDash Engineering

MARCH 27, 2024

Real-time event processing is a critical component of a distributed system’s scalability. At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. One of the challenges we face, however, is how to properly validate the system before going live.

Kafka 103

Kafka Architecture Algorithm Systems 103

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system. Building the KPay payment system.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Confluent

JUNE 26, 2019

Microservices have a symbiotic relationship with domain-driven design (DDD)—a design approach where the business domain is carefully modeled in software and evolved over time, independently of the plumbing that makes the system work. I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®.

Kafka 109

Kafka Designing Architecture Coding 109

Monte Carlo

NOVEMBER 12, 2024

While not every company needs to process millions of events per second, understanding these advanced architectures helps us make better decisions about our own data infrastructure, whether we’re handling user recommendations, ride-sharing logistics, or simply figuring out which meeting rooms are actually being used.

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. And if you weren’t there? Well, dig in and start learning! 3 keynotes.

Kafka 109

Kafka Cloud Database Engineering 109

Confluent

JANUARY 7, 2021

At Zendesk, Apache Kafka® is one of our foundational services for distributing events among different internal systems. We have pods, which can be thought of as isolated cloud environments where […].

Kafka 119

Kafka Cloud Systems 119

Snowflake

MARCH 2, 2023

But before data can be transformed and served or shared, it must be ingested from source systems. As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. This solution is both scalable and reliable, as we have been able to effortlessly ingest upwards of 1GB/s throughput.”

Kafka 128

Kafka Data Ingestion Data Pipeline Cloud Storage 128

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Confluent

MARCH 29, 2021

Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and […].

Kafka 131

Kafka Systems Management IT 131

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Confluent

APRIL 16, 2019

It seems like there’s a Kafka Summit every other month. We now have the Kafka Summit New York in the books, and the session videos are available in record time. As I usually do, let me break the event down for you. We planned the New York event to be a bit smaller than last fall’s flagship San Francisco Summit.

Kafka 102

Kafka Database Programming Systems 102

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Medical Database 21

Confluent

APRIL 11, 2019

This explains why users have been looking for a reliable way to stream their data from Apache Kafka ® to S3 since Kafka Connect became available. In March 2017, we released the Kafka Connect S3 connector as part of the Confluent Platform. And no one likes missing events. So, it happened. How about we take it for a spin?

Kafka 110

Kafka AWS Metadata Architecture 110

Workfall

JUNE 27, 2023

It means that there is a high risk of data loss but Apache Kafka solves this because it is distributed and can easily scale horizontally and other servers can take over the workload seamlessly. This is where Apache Kafka comes in. Kafka can also be used to stream data from IoT devices or sensors. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. However, Apache Kafka is more than just messaging. Some Kafka and Rockset users have also built real-time e-commerce applications , for example, using Rockset’s Java, Node.js

Kafka 21

Kafka SQL BI Hadoop 21

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®. and OpenID Connect.

Kafka 95

Kafka Cloud Data Pipeline PostgreSQL 95

Confluent

MAY 21, 2021

Apache Kafka® applications run in a distributed manner across multiple containers or machines. And in the world of distributed systems, what can go wrong often goes wrong. This blog post […].

Kafka 128

Kafka Systems 128

Confluent

MAY 28, 2020

For many organizations, Apache Kafka® is the backbone and source of truth for data systems across the enterprise. Protecting your event streaming platform is critical for data security and often […].

Kafka 113

Kafka Data Security Systems Data 113

Confluent

MAY 7, 2019

Every developer who uses Apache Kafka ® has used a Kafka consumer at least once. Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. Consistency.

Kafka 111

Kafka Accessible Accessibility Metadata 111

Confluent

MARCH 20, 2019

Back in May 2017, we laid out why we believe that Kafka Streams is better off without a concept of watermarks or triggers , and instead opts for a continuous refinement model. By continuous refinement , I mean that Kafka Streams emits new results whenever records are updated. They are all non-retractable. Use case: Alerting.

Kafka 106

Kafka Programming Process Designing 106

Data Engineering Podcast

JULY 6, 2020

Summary There are an increasing number of use cases for real time data, and the systems to power them are becoming more mature. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home.

Systems 100

Systems Kafka SQL Government 100

Confluent

JUNE 13, 2019

This data pipeline is a great example of a use case for Apache Kafka ®. Observational astronomers study many different types of objects, from asteroids in our own solar system to galaxies that are billions of lightyears away. The case for Apache Kafka. Astronomy in real time.

Kafka 102

Kafka Bytes Data Pipeline Python 102

Confluent

MAY 13, 2019

In the last year, we’ve experienced enormous growth on Confluent Cloud, our fully managed Apache Kafka ® service. As Confluent Cloud has grown, we’ve noticed two gaps that very clearly remain to be filled in managed Apache Kafka services. Five seconds to Kafka (or, never make another cluster again!).

Kafka 100

Kafka Cloud Building Management 100

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

JUNE 27, 2019

Confluent’s clients for Apache Kafka ® recently passed a major milestone—the release of version 1.0. Magnus Edenhill first started developing librdkafka about seven years ago, later joining Confluent in the very early days to help foster the community of Kafka users outside the Java ecosystem. In the v0.11.6

Kafka 106

Kafka Designing Java Coding 106

Confluent

SEPTEMBER 30, 2019

Go from zero to production on Apache Kafka ® without talking to sales reps or building infrastructure. Apache Kafka is the standard for event-driven applications. Free Kafka as a service. Kafka made serverless. Grow as you go based on load, as your underlying Kafka infrastructure scales up and down transparently.

Kafka 19

Kafka Cloud Amazon Web Services Google Cloud 19

Confluent

OCTOBER 16, 2019

Trains are an excellent source of streaming data—their movements around the network are an unbounded series of events. Using this data, Apache Kafka ® and Confluent Platform can provide the foundations for both event-driven applications as well as an analytical platform. As with any real system, the data has “character.”

Kafka 19

Kafka Building Data Coding 19

Confluent

AUGUST 13, 2019

Yes, the other definition of event sourcing. At Kafka Summit SF, you’ll get to hear incredible keynotes from leading technologists, including Jay Kreps and Neha Narkhede , original co-creators of Apache Kafka ®. Nowhere else in the world will you find a larger concentration of Kafka community genius than here. Get it, PR?

Kafka 19

Kafka Food Electronics Recruitment 19