Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 137

Kafka Data Ingestion Data Pipeline Cloud Storage 137

Confluent

JUNE 26, 2020

We are excited to announce the preview release of the fully managed Snowflake sink connector in Confluent Cloud, our fully managed event streaming service based on Apache Kafka®. Our managed […].

Kafka 105

Kafka Cloud Management 105

Confluent

MAY 30, 2022

What we’ve done to evolve from cloud Kafka to Confluent Cloud, a data streaming platform that’s 10X better than Kafka in elasticity, storage, resiliency, and more.

Kafka 95

Kafka Cloud Data 95

Confluent

OCTOBER 19, 2021

Extensive out-of-the-box functionality, a large user community, and up-to-date, cloud-native features make Spring and its libraries a strong option for anchoring your Apache Kafka® and Confluent Cloud based microservices architecture. […].

Kafka 130

Kafka Architecture Cloud IT 130

Confluent

MARCH 9, 2022

Within a few minutes, you provision a fully managed Apache Kafka® cluster […]. Imagine your team wants to design a data streaming architecture and you’re in charge of creating the prototype.

Kafka 126

Kafka Cloud Architecture Designing 126

Confluent

FEBRUARY 18, 2020

Less than six months ago, we announced support for Microsoft Azure in Confluent Cloud, which allows developers using Azure as a public cloud to build event streaming applications with Apache […].

Cloud 145

Cloud Kafka Building Programming 145

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Confluent

MAY 23, 2023

Learn how Confluent Cloud helped Michelin streamline Apache Kafka® operations, reduce costs, and go to market 8-9 months faster. Keyword: Confluent Cloud

Kafka 96

Kafka Cloud 96

Confluent

JANUARY 7, 2021

At Zendesk, Apache Kafka® is one of our foundational services for distributing events among different internal systems. We have pods, which can be thought of as isolated cloud environments where […].

Kafka 119

Kafka Cloud Systems 119

Confluent

SEPTEMBER 1, 2020

If you are taking your first steps with Apache Kafka®, looking at a test environment for your client application, or building a Kafka demo, there are two “easy button” paths […].

Kafka 133

Kafka Building Cloud 133

Confluent

MAY 9, 2023

Low utilization and operational complexity dramatically increases Kafka costs, so we reinvented Kafka as a cloud-native and complete service to reduce costs for thousands of businesses at any scale.

Kafka 107

Kafka Utilities Cloud 107

Confluent

SEPTEMBER 9, 2024

Confluent has acquired WarpStream, an innovative Kafka-compatible streaming solution. Read the full statement by Jay Kreps, co-founder and CEO of Confluent.

Cloud 142

Cloud Kafka 142

Cloudera

OCTOBER 12, 2020

We are thrilled to announce that Cloudera has acquired Eventador , a provider of cloud-native services for enterprise-grade stream processing. The DataFlow platform has established a leading position in the data streaming market by unlocking the combined value and synergies of Apache NiFi, Apache Kafka and Apache Flink.

Cloud 132

Cloud Process Scala Kafka 132

DoorDash Engineering

MARCH 27, 2024

At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka. In Kafka, a test tenant processing production event can cause data inconsistencies, including outages and other incidents.

Kafka 103

Kafka Architecture Algorithm Process 103

Confluent

JULY 1, 2020

This is the third month of Project Metamorphosis, where we discuss new features in Confluent’s offerings that bring together event streams and the best characteristics of modern cloud data systems. […].

Project 122

Project Cloud Kafka Systems 122

Confluent

OCTOBER 7, 2020

The cloud opens up exciting new opportunities for information gathering, analysis, and sharing that can make every organization’s products and services better. Thanks to the cloud and its decentralized nature, […].

Cloud 111

Cloud Project Kafka IT 111

Confluent

JULY 27, 2021

Serverless offerings in the cloud are a favorite among software engineers—a prime example are object stores such as AWS S3. For the system designer, however, it is an engineering challenge […].

Cloud 105

Cloud Kafka Software Engineering Software Engineer 105

Confluent

JANUARY 10, 2024

Confluent Cloud is now 10x faster than Apache Kafka. Read our latency benchmarking results, the innovations behind-the-scenes, and the lessons we learned.

Kafka 89

Kafka Cloud 89

Confluent

JULY 28, 2021

At Confluent, we focus on the holy trinity of performance, price, and availability, with the goal of delivering a similar performance envelope for all workloads across all supported cloud providers. […].

Cloud 122

Cloud Kafka Architecture 122

Confluent

JUNE 17, 2024

Improve REST API response data w/Kafka and Flink SQL in Confluent Cloud; Automatic connector retriability combats REST flakiness; Demo w/OpenSky data.

Kafka 105

Kafka SQL Cloud Data 105

Confluent

SEPTEMBER 27, 2021

The distributed architecture of Apache Kafka® can cause the operational burden of managing it to quickly become a limiting factor for adoption and developer agility. For this reason, it is […].

Kafka 97

Kafka AWS Cloud Architecture 97

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Snowflake

NOVEMBER 11, 2024

With Snowpipe for Apache Kafka (public preview soon in AWS and Microsoft Azure), a “pull” mechanism, rather than the existing “push” connector, allows you to extract and ingest Apache Kafka events into your Snowflake account directly without hosting your own Kafka Connect cluster.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Jesse Anderson

JUNE 12, 2024

We also discuss the various systems using Kafka’s protocol. Confluent has never shied away from saying Kafka is “easy,” and I disagree. During the Kafka Summit London Keynote, the speakers said “easy” 17 times; in the Kafka Summit Bangalore Keynote, they said it 18 times. Using Confluent Cloud?

Kafka 147

Kafka Data Lake Architecture Cloud 147

Confluent

JULY 15, 2020

With billions of Internet of Things (IoT) devices, achieving real-time interoperability has become a major challenge. Together, Confluent, Waterstream, and MQTT are accelerating Industry 4.0 with new Industrial IoT (IIoT) […].

Kafka 139

Kafka Cloud Programming 139

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. Lastly, I share my experience implementing a similar pipeline on the Google Cloud Platform. A table is a distributed in-memory dictionary, backed by a Kafka changelog topic.

Kafka 98

Kafka Python Process Google Cloud 98

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters. Built atop Prometheus.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

MARCH 11, 2022

With the launch of CDP Public Cloud 7.2.14, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! In this release , the Streams Messaging templates in Data Hub will come with Apache Kafka 2.8 Kafka & Cruise Control Updates. Kafka Updates: Deployments with Kafka 2.5

Cloud 107

Cloud Kafka Utilities Database 107

Confluent

JANUARY 8, 2020

This article show how you can offload data from on-premises transactional (OLTP) databases to cloud-based datastores, including Snowflake and Amazon S3 with Athena. I’m also going to take the opportunity […].

Cloud 27

Cloud Database Data Kafka 27

Cloudera

FEBRUARY 10, 2022

After the launch of Cloudera DataFlow for the Public Cloud (CDF-PC) on AWS a few months ago, we are thrilled to announce that CDF-PC is now generally available on Microsoft Azure, allowing NiFi users on Azure to run their data flows in a cloud-native runtime. . The need for a cloud-native Apache NiFi service on Microsoft Azure.

Cloud 116

Cloud Kafka AWS Data Ingestion 116

Confluent

MARCH 19, 2021

LiveStreams is a YouTube show about Confluent, real-time data streaming, and related technologies that help you maximize data in motion on any cloud. Every episode of LiveStreams will teach you […].

Cloud 98

Cloud Process Kafka Technology 98

Confluent

JULY 30, 2021

Last year, Confluent announced support for Infinite Storage, which fundamentally changes data retention in Apache Kafka® by allowing […]. It’s about maintaining the right data even when no one is watching.

Data Integration 136

Data Integration Cloud Kafka Data 136

Cloudera

AUGUST 10, 2021

With the release of CDP Private Cloud (PvC) Base 7.1.7, Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Deep Dive 2: Atlas / Kafka integration. Deep Dive 1: Impala Row filtering.

Cloud 97

Cloud Kafka Metadata SQL 97

Confluent

MAY 6, 2020

A few weeks ago when we talked about our new fundraising, we also announced we’d be kicking off Project Metamorphosis. What is Project Metamorphosis? Let me try to explain. I […].

Project 124

Project Kafka Cloud 124

Netflix Tech

APRIL 8, 2025

In cloud environments, IP addresses are reassigned to different workloads as workload instances are created and terminated, so IP addresses alone cannot provide insights on which workloads are communicating. We implemented a broadcasting mechanism using Kafka, where each node publishes learned time ranges to all other nodes.

AWS 75

AWS Kafka Cloud Programming 75

Confluent

JUNE 7, 2022

How to elastically scale Kafka clusters from 0 to 100 MB/s and back with automatic cluster resizing, data rebalancing, real-time consumption optimization, and monitoring in seconds.

Kafka 81

Kafka Cloud Data 81

Cloudera

AUGUST 16, 2021

Are you ready to turbo-charge your data flows on the cloud for maximum speed and efficiency? Deploy, manage and monitor your standard NiFi flows running on-premises or on CDP Data Hub into cloud-native flows running on Kubernetes clusters on AWS. . NEW Cloudera DataFlow for the Public Cloud.

Cloud 112

Cloud AWS Kafka Utilities 112