Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

Confluent

APRIL 23, 2019

When it was first created, Apache Kafka ® had a client API for just Scala and Java. Since then, the Kafka client API has been developed for many other programming languages which enables you to pick the language you want. At Confluent, we have an engineering team dedicated to the development of these Kafka clients.

Programming Language 102

Programming Language Kafka Programming Scala 102

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This article describes how to instrument Kafka-based applications with distributed tracing capabilities in order to make dataflows between event-based components more visible.

Kafka 111

Kafka Transportation Metadata Consulting 111

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. However, Apache Kafka is more than just messaging. Some Kafka and Rockset users have also built real-time e-commerce applications , for example, using Rockset’s Java, Node.js

Kafka 21

Kafka SQL BI Hadoop 21

Confluent

JUNE 27, 2019

Confluent’s clients for Apache Kafka ® recently passed a major milestone—the release of version 1.0. Magnus Edenhill first started developing librdkafka about seven years ago, later joining Confluent in the very early days to help foster the community of Kafka users outside the Java ecosystem. Leading up to the 1.0

Kafka 106

Kafka Designing Java Coding 106

Confluent

FEBRUARY 12, 2019

One of the most common integrations that people want to do with Apache Kafka ® is getting data in from a database. The existing data in a database, and any changes to that data, can be streamed into a Kafka topic. Here, I’m going to dig into one of the options available—the JDBC connector for Kafka Connect. Introduction.

Kafka 90

Kafka MySQL Bytes Java 90

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Medical Database 21

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. We used Groovy instead of Java to write our UDFs, so we’ve applied the groovy plugin. gradlew composeUp. Note: When executing./gradlew

Kafka 89

Kafka Java Bytes SQL 89

Confluent

JULY 24, 2019

Using Jaeger tracing, I’ve been able to answer an important question that nearly every Apache Kafka ® project that I’ve worked on posed: how is data flowing through my distributed system? Distributed tracing with Apache Kafka and Jaeger. Example of a Kafka project with Jaeger tracing. What does this all mean?

Kafka 54

Kafka Systems Bytes Project 54

Confluent

NOVEMBER 29, 2022

Apache Kafka’s Streams API embeds Machine Learning into any app or microservice (Java, Docker, Kubernetes, etc.) to add business value.

Machine Learning 145

Machine Learning Kafka Java Building 145

Data Engineering Podcast

MARCH 2, 2020

The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing. Developers can work with the SQL constructs that they are familiar with while automatically getting the durability and reliability that Kafka offers. How is ksqlDB architected?

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

MAY 8, 2019

In anything but the smallest deployment of Apache Kafka ® , there are often going to be multiple clusters of Kafka Connect and KSQL. Kafka Connect rebalances when connectors are added/removed, and this can impact the performance of other connectors on the same cluster. Streaming data into Kafka with Kafka Connect.

Kafka 89

Kafka Management Hadoop Database 89

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Testing the Kafka Connect Neo4j Sink. curl -X POST [link]. jar -f AVRO -e 100000.

Kafka 88

Kafka Java Programming Language Data Pipeline 88

Confluent

AUGUST 1, 2019

The Kafka Streams API boasts a number of capabilities that make it well suited for maintaining the global state of a distributed system. At Imperva, we took advantage of Kafka Streams to build shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in our system.

Kafka 20

Kafka Systems Building Metadata 20

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Microservices, Apache Kafka, and Domain-Driven Design (DDD) covers this in more detail. Example: Severstal. High throughput.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®. Command Line Shell.

Kafka 95

Kafka Cloud Data Pipeline PostgreSQL 95

Christophe Blefari

MARCH 15, 2024

Obviously Benoit prefers Kestra, at the expense of writing YAML and running a Java application. Unlocking Kafka's potential: tackling tail latency with eBPF. New Apache Arrow engines — Arrow has become one of the most used library when it comes to built in-memory engines.

Metadata 272

Metadata Data Datasets Data Warehouse 272

Confluent

APRIL 10, 2019

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 87

Kafka Java Metadata Cloud 87

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Confluent

OCTOBER 7, 2019

This tutorial describes how to set up a sample Spring Boot application in Pivotal Application Service (PAS), which consumes and produces events to an Apache Kafka ® cluster running in Pivotal Container Service (PKS). With this tutorial, you can set up your PAS and PKS configurations so that they work with Kafka. Methodology.

Kafka 18

Kafka Java Coding Accessible 18

Confluent

OCTOBER 16, 2019

Using this data, Apache Kafka ® and Confluent Platform can provide the foundations for both event-driven applications as well as an analytical platform. With tools like KSQL and Kafka Connect, the concept of streaming ETL is made accessible to a much wider audience of developers and data engineers. Ingesting the data.

Kafka 19

Kafka Building Data Coding 19

Confluent

OCTOBER 23, 2019

If you’ve worked with the Apache Kafka ® and Confluent ecosystem before, chances are you’ve used a Kafka Connect connector to stream data into Kafka or stream data out of it. This article will cover the basic concepts and architecture of the Kafka Connect framework. What is Kafka Connect?

Kafka 15

Kafka Cloud Storage Cloud Database 15

Confluent

MAY 2, 2019

When managing Apache Kafka ® clusters at scale, tasks that are simple on small clusters turn into significant burdens. Relatedly, KIP-226 enabled dynamic broker reconfiguration since Apache Kafka 1.1. See the documentation (or, if you please, the Apache Kafka wiki ) for a complete list of which parameters this applies to.

Kafka 89

Kafka Management Food Consulting 89

Confluent

SEPTEMBER 5, 2019

Following on from How to Work with Apache Kafka in Your Spring Boot Application , which shows how to get started with Spring Boot and Apache Kafka ® , here I will demonstrate how to enable usage of Confluent Schema Registry and Avro serialization format in your Spring Boot applications. Initial revision. Prerequisities. Avro SerDes.

Kafka 20

Kafka Java Food Cloud 20

Confluent

JUNE 11, 2019

Here in part 4 of the Spring for Apache Kafka Deep Dive blog series, we will cover: Common event streaming topology patterns supported in Spring Cloud Data Flow. Create and manage event streaming pipelines, including a Kafka Streams application using Spring Cloud Data Flow. java -jar spring-cloud-dataflow-shell-2.1.0.RELEASE.jar.

Kafka 86

Kafka Cloud Java MongoDB 86

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Confluent

SEPTEMBER 14, 2023

Learn how to use Apache Flink to build a Java pipeline that consumes clickstream data from Apache Kafka.

Kafka 72

Kafka Programming Java Building 72

Cloudera

FEBRUARY 2, 2021

In the previous posts in this series, we have discussed Kerberos , LDAP and PAM authentication for Kafka. In this post we will look into how to configure a Kafka cluster and client to use a TLS client authentication. TLS is assumed to be enabled for the Apache Kafka cluster, as it should be for every secure cluster.

Kafka 80

Kafka Certification Java Government 80

Cloudera

DECEMBER 2, 2020

This is the first installment in a short series of blog posts about security in Apache Kafka. Secured Apache Kafka clusters can be configured to enforce authentication using different methods, including the following: SSL – TLS client authentication. We use the kafka-console-consumer for all the examples below.

Kafka 70

Kafka Java Big Data Ecosystem Cloud 70

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. Given that around 50% of Java clients at Pinterest are on Flink, PSC integration with Flink was key to achieving our platform goals of fully migrating Java clients to PSC.

Kafka 110

Kafka Java Software Engineer Software Engineering 110

Confluent

APRIL 20, 2020

The goal of this post is to illustrate PUSH to web from Apache Kafka® with a hands-on example. Our business users are always wanting their data faster so they can […].

Kafka 94

Kafka Java Data 94

Data Engineering Podcast

OCTOBER 15, 2023

What was the process for adding full Java support in addition to SQL? What was the process for adding full Java support in addition to SQL? What are the problems that customers are trying to solve when they come to Decodable? When you launched your focus was on SQL transformations of streaming data.

Process 182

Process Building SQL BI 182

Rock the JVM

JANUARY 21, 2023

Introduction Apache Kafka is a well-known event streaming platform used in many organizations worldwide. The focus of this article is to provide a better understanding of how Kafka works under the hood to better design and tune your client applications. Environment Setup First, we want to have a Kafka Cluster up and running.

Kafka 69

Kafka Java Scala Coding 69

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. gradlew composeUp. The KSQL pipeline flow.

Kafka 96

Kafka Management Bytes SQL 96

Cloudera

AUGUST 2, 2021

Since all the flows were simple event processing, the NiFi flows were built out in a matter of hours (drag-and-drop) instead of months (coding in Java). . They asked, “Can NiFi keep up with the same throughput as Storm?” Setting the context, why would a customer want to use Apache NiFi, Apache Kafka, and Apache HBase? Nifi Flows.

Kafka 120

Kafka Java Coding Process 120

Data Engineering Weekly

JULY 28, 2024

JSON workflow definition gives flexibility to build DSL on higher-level languages like Python & Java. link] Uber: Introduction to Kafka Tiered Storage at Uber The effectiveness of Kafka Tiered-Storage is a widely discussed topic. A key highlight for me is the following features from Maestro. Pipeline breakpoint feature.

Data Engineering 111

Data Engineering Data Engineer Engineering Database-centric 111

Confluent

APRIL 2, 2019

Includes free forever Confluent Platform on a single Apache Kafka ® broker, improved Control Center functionality at scale and hybrid cloud streaming. the event streaming platform built by the original creators of Apache Kafka. in order to bring our C/C++, Python, Go and.NET clients closer to parity with the Java client.

Kafka 21

Kafka Java Cloud Metadata 21