Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. However, Apache Kafka is more than just messaging. Some Kafka and Rockset users have also built real-time e-commerce applications , for example, using Rockset’s Java, Node.js

Kafka 21

Kafka SQL BI Hadoop 21

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Database Medical 21

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®. Command Line Shell.

Kafka 95

Kafka Cloud Data Pipeline PostgreSQL 95

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. We used Groovy instead of Java to write our UDFs, so we’ve applied the groovy plugin. gradlew composeUp. Note: When executing./gradlew

Kafka 90

Kafka Java Bytes SQL 90

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Microservices, Apache Kafka, and Domain-Driven Design (DDD) covers this in more detail. Example: Severstal.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

Confluent

AUGUST 1, 2019

The Kafka Streams API boasts a number of capabilities that make it well suited for maintaining the global state of a distributed system. At Imperva, we took advantage of Kafka Streams to build shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in our system.

Kafka 20

Kafka Systems Building Metadata 20

Confluent

APRIL 10, 2019

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 88

Kafka Java Metadata Cloud 88

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 111

Kafka Java Software Engineer Software Engineering 111

Confluent

JUNE 11, 2019

Here in part 4 of the Spring for Apache Kafka Deep Dive blog series, we will cover: Common event streaming topology patterns supported in Spring Cloud Data Flow. Create and manage event streaming pipelines, including a Kafka Streams application using Spring Cloud Data Flow. java -jar spring-cloud-dataflow-shell-2.1.0.RELEASE.jar.

Kafka 87

Kafka Cloud Java MongoDB 87

Data Engineering Weekly

JULY 28, 2024

The blog is an excellent summarization of the common patterns emerging in GenAI platforms. The blog Prompt Engineering for a Better SQL Code Generation With LLMs is a pretty good guide on applying prompt engineering to improve productivity. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Cloudera

AUGUST 2, 2021

Since all the flows were simple event processing, the NiFi flows were built out in a matter of hours (drag-and-drop) instead of months (coding in Java). . As you’ll see in this blog, NiFi is not only keeping up with Storm; it beats Storm by 4x throughput. . Nifi Flows. Take the next steps to: Learn about Cloudera Flow Management.

Kafka 119

Kafka Java Coding Process 119

Cloudera

DECEMBER 2, 2020

This is the first installment in a short series of blog posts about security in Apache Kafka. Secured Apache Kafka clusters can be configured to enforce authentication using different methods, including the following: SSL – TLS client authentication. We use the kafka-console-consumer for all the examples below.

Kafka 69

Kafka Java Big Data Ecosystem Cloud 69

Cloudera

FEBRUARY 2, 2021

In the previous posts in this series, we have discussed Kerberos , LDAP and PAM authentication for Kafka. In this post we will look into how to configure a Kafka cluster and client to use a TLS client authentication. TLS is assumed to be enabled for the Apache Kafka cluster, as it should be for every secure cluster.

Kafka 79

Kafka Certification Java Management 79

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. Sample repository. gradlew composeUp.

Kafka 97

Kafka Management Bytes SQL 97

Confluent

MAY 28, 2019

Part 1 of this blog series by Gwen Shapira explained the benefits of schemas, contracts between services, and compatibility checking for schema evolution. Actually, we recommend that you consider another alternative to self-managing Schema Registry, and the next blog post in this series reveals what that alternative is! Bad designs.

Management 85

Management Kafka Java Certification 85

Confluent

JUNE 18, 2019

Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®. Building an indexing pipeline at scale with Kafka Connect. Moving data into Apache Kafka with the JDBC connector. For this use case, we are going to use it as a source connector.

Architecture 97

Architecture Building Kafka Database-centric 97

Confluent

SEPTEMBER 6, 2019

In the above scenario, we would have to update all five services to connect to Apache Kafka ® , create the event in all the appropriate places inside each service, and then produce that event to a Kafka topic. The first point to consider is if connecting to Kafka from all services is even possible.

Kafka 22

Kafka MySQL Banking Java 22

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion.

Process 91

Process Kafka Scala SQL 91

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection. Kafka REST Proxy for streaming data.

Data Pipeline 94

Data Pipeline Building Kafka Java 94

Confluent

JULY 3, 2019

The idea in this blog post is to mix information coming from two distinct channels: the RSS feeds of sport-related newspapers and Twitter feeds of the FIFA Women’s World Cup. Ingesting Twitter data is very easy with Kafka Connect , a framework for connecting Kafka with external systems. Ingesting Twitter data. connector.state].

Data Analysis 21

Data Analysis Kafka Datasets Java 21

Christophe Blefari

JUNE 21, 2024

you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with I won't delve into every announcement here, but for more details, SELECT has written a blog covering the 28 announcements and takeaways from the Summit. Accordingly to the press Snowflake and Confluent (Kafka) were also trying to buy Tabular.

Metadata 147

Metadata Data Warehouse BI MySQL 147

LinkedIn Engineering

OCTOBER 19, 2023

In 2010, they introduced Apache Kafka , a pivotal Big Data ingestion backbone for LinkedIn’s real-time infrastructure. To transition from batch-oriented processing and respond to Kafka events within minutes or seconds, they built an in-house distributed event streaming framework, Apache Samza. hours to 25 minutes).

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Jesse Anderson

DECEMBER 12, 2022

Apache Kafka came in 2011 and gave the industry a much better way to move real-time data. Apache Kafka has its architectural limitations, and Apache Pulsar was released in 2016. At various times it’s been Java, Scala, and Python. Apache Flink came in 2011 and gave us our first real streaming engine.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

ProjectPro

JUNE 18, 2021

Why do data scientists prefer Python over Java? Java vs Python for Data Science- Which is better? Which has a better future: Python or Java in 2021? This blog aims to answer all questions on how Java vs Python compare for data science and which should be the programming language of your choice for doing data science in 2021.

Java 52

Java Data Science Python Programming Language 52

Cloudera

JANUARY 19, 2022

Some of these improvements and features are: Flink JAR submission (for Java UDF’s). Run on your desktop or cloud node, connecting to Kafka or other sources/sinks via API calls to their respective clusters. We plan future blog posts on this workflow. Release Notes appeared first on Cloudera Blog. x compatibility.

Java 90

Java Kafka SQL Cloud 90

Confluent

MAY 21, 2019

The profile service will publish the changes in profiles, including address changes to an Apache Kafka ® topic, and the quote service will subscribe to the updates from the profile changes topic, calculate a new quote if needed and publish the new quota to a Kafka topic so other services can subscribe to the updated quote event.

Kafka 110

Kafka Insurance Architecture Database 110

Netflix Tech

MARCH 10, 2023

Data Sharding strategy in elasticsearch is updated to provide low search latency (as described in blog post) Design of new Cassandra reverse indices to support different sets of queries. For asynchronous processing, events are sent to Apache Kafka topics to be processed. Apache Kafka topic is configured as a message broker.

Management 97

Management Kafka Metadata Media 97

Cloudera

APRIL 20, 2021

The developers must understand lower-level languages like Java and Scala and be familiar with the streaming APIs. Streamings Messaging , powered by Apache Kafka, buffers and scales massive volumes of data streams for streaming analytics. appeared first on Cloudera Blog. Take the next step and learn more: Cloudera DataFlow.

Kafka 95

Kafka Hospitality Retail Data Ingestion 95

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Real-time Ingestion Events from our real-time analytics pipeline were configured to be sent into our internal Flink application, streamed to Kafka, and written into Druid.

Kafka 106

Kafka Data Ingestion Datasets Architecture 106

Confluent

AUGUST 6, 2019

The previous blog post How to Build a UDF and/or UDAF in KSQL 5.0 creating custom KSQL functions is even easier when you leverage Maven , a tool for building and managing dependencies in Java projects. java ? ? ??? For this example, we’ll assume there’s a topic named api_logs in our Kafka cluster. my-udf/ ???

Kafka 18

Kafka Java Coding Project 18

Cloudera

FEBRUARY 22, 2023

In SSB, today we are supporting JavaScript (JS) and Java UDFs, which can be used as a function with your data. But let’s assume we have already set up such a table, based off of a Kafka topic that has the ADSB data streaming through it, and we have named it airplanes. With UDFs you can really enhance the capabilities of your queries.

SQL 85

SQL Raw Data Kafka Programming Language 85

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Cloudera

JUNE 28, 2022

In a previous blog of this series, Turning Streams Into Data Products , we talked about the increased need for reducing the latency between data generation/ingestion and producing analytical results and insights from this data. This blog will be published in two parts. This is what we call the first-mile problem. The use case.

Process 85

Process Kafka SQL Machine Learning 85

Confluent

FEBRUARY 27, 2019

When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. By persisting the streams in Kafka we then have a record of all system activity (a source of truth), and also a mechanism to drive reactions. Event-first FaaS.

Kafka 109

Kafka AWS Architecture Cloud 109

Cloudera

FEBRUARY 9, 2021

If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database. To know more about Apache HBase region splitting and merging, see the blog post here: [link].

Database 104

Database Java SQL Data Ingestion 104