Knowledge Hut

MAY 3, 2024

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. The demand for stream processing is increasing every day in today’s era. Kafka streams provide true a-record-at-a-time processing capabilities.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Confluent

MARCH 26, 2024

In this guide, you will learn how to track events in a large codebase like GitHub using Apache Kafka and Kafka Streams.

Kafka 69

Kafka Process Data 69

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

MARCH 11, 2020

In the Apache Kafka® ecosystem, ksqlDB and Kafka Streams are two popular tools for building event streaming applications that are tightly integrated with Apache Kafka. While ksqlDB and Kafka Streams […].

Kafka 115

Kafka Process Building 115

DoorDash Engineering

AUGUST 13, 2024

This journey began with Kafka, one of our most critical and widely used infrastructure components. Kafka is a distributed event streaming platform that DoorDash uses to handle billions of real-time events. To address this, we developed Kafka Self-Serve, our flagship self-serve storage infrastructure platform.

Kafka 82

Kafka Engineering AWS Designing 82

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Confluent

NOVEMBER 30, 2022

Putting Apache Kafka To Use: A Practical Guide to Building an Event Streaming Platform.

Kafka 105

Kafka Building Process 105

Data Engineering Podcast

OCTOBER 15, 2023

Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. Check out the agenda and register today at Neo4j.com/NODES.

Process 182

Process Building SQL BI 182

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. FROM SYSMON_JOIN.

Process 83

Process Kafka SQL Datasets 83

Confluent

JUNE 11, 2019

For event streaming application developers, it is important to continuously update the streaming pipeline based on the need for changes in the individual applications in the pipeline. It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline.

Kafka 86

Kafka Cloud Java MongoDB 86

Netflix Tech

AUGUST 1, 2022

A Data Movement and Processing Platform @ Netflix By Bo Lei , Guilherme Pires , James Shao , Kasturi Chatterjee , Sujay Jain , Vlad Sydorenko Background Realtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users.

Process 109

Process Transportation Kafka Entertainment 109

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 126

Kafka Data Ingestion Data Pipeline Cloud Storage 126

Cloudera

AUGUST 10, 2022

Cloudera has a strong track record of providing a comprehensive solution for stream processing. Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. Cloudera Stream Processing Community Edition. Apache Kafka and SMM.

Process 89

Process Kafka PostgreSQL MySQL 89

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Rock the JVM

JANUARY 21, 2023

Giannis is a proud alumnus of Rock the JVM, working as a Solutions Architect with a focus on Event Streaming and Stream Processing Systems. Introduction Apache Kafka is a well-known event streaming platform used in many organizations worldwide. Environment Setup First, we want to have a Kafka Cluster up and running.

Kafka 69

Kafka Java Scala Coding 69

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. Four pillars of event streaming. Pillar 1 – Business function: Payment processing pipeline. Pillar 4 – Operational plane: Event logging, DLQs and automation. Journey to Event Driven – Part 2: Programming Models for the Event-Driven Architecture.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Confluent

SEPTEMBER 24, 2020

Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring […].

Kafka 143

Kafka Engineering Systems Process 143

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. Event-first FaaS? FaaS for streaming processing.

Kafka 109

Kafka AWS Architecture Cloud 109

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. Similar to Google in web browsing and Photoshop in image processing, it became a gold standard in data streaming, preferred by 70 percent of Fortune 500 companies. What is Kafka? What Kafka is used for.

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Confluent

SEPTEMBER 29, 2020

ksqlDB, the event streaming database, is becoming one of the most popular ways to work with Apache Kafka®. Every day, we answer many questions about the project, but here’s a […].

Process 145

Process Kafka Database Project 145

LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Confluent

NOVEMBER 30, 2022

This is the second part of our guide on streaming data and Apache Kafka. This guide will contain specific advice on how to go about building an event streaming platform in your organization.

Kafka 52

Kafka Building Data Process 52

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Healthcare Medical 52

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 93

Kafka Technology IT Government 93

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Being event first.

Architecture 92

Architecture Programming Kafka Database-centric 92

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 78

Kafka Data Cloud Big Data 78

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Decoupling is the process of organizing a system around business capabilities to form an architecture that is decentralized. Microservices.

Kafka 109

Kafka Designing Architecture Coding 109

Rockset

MARCH 5, 2020

Apache Kafka has seen broad adoption as the streaming platform of choice for building applications that react to streams of data in real time. In many organizations, Kafka is the foundational platform for real-time event analytics, acting as a central location for collecting event data and making it available in real time.

Kafka 40

Kafka Data Warehouse Data Lake Relational Database 40

Knowledge Hut

JANUARY 18, 2024

“Big data Analytics” is a phrase that was coined to refer to amounts of datasets that are so large traditional data processing software simply can’t manage them. Using these high-tech tools, I can explore economic trends, figure out patterns, and predict future events. It has changed how we look at and understand data.

Data Process 52

Data Process Process Hadoop Scala 52

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. You can use this as an example of how to add graph abilities to any event streaming application.

Kafka 55

Kafka Process Algorithm Cloud 55

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. And if you weren’t there? Well, dig in and start learning! 3 keynotes.

Kafka 109

Kafka Cloud Database Engineering 109

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®. and OpenID Connect.

Kafka 95

Kafka Cloud Data Pipeline PostgreSQL 95

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Cloudera

SEPTEMBER 20, 2021

There are many ways that Apache Kafka has been deployed in the field. In our Kafka Summit 2021 presentation, we took a brief overview of many different configurations that have been observed to date. Kafka as software falls more cleanly into the Parallel Systems Reliability discussed below but some parts of it can end up Serial.

Kafka 114

Kafka Systems Utilities Bytes 114

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Cloudera

JULY 18, 2022

In this blog we will conclude the implementation of our fraud detection use case and understand how Cloudera Stream Processing makes it simple to create real-time stream processing pipelines that can achieve neck-breaking performance at scale. Flink is a “streaming first” modern distributed system for data processing.

Process 89

Process Kafka Scala SQL 89