Confluent

AUGUST 11, 2020

However, not all databases can be in the […]. Building a Cloud ETL Pipeline on Confluent Cloud shows you how to build and deploy a data pipeline entirely in the cloud.

Database 103

Database Kafka Data Pipeline Cloud 103

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. Privacy Engineering for the World of Kafka by Alexander Cook of Privitar.

Kafka 109

Kafka Cloud Database Engineering 109

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineering 50

Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Database Medical 21

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on. Here we go!

Kafka 55

Kafka Process Algorithm Cloud 55

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This article describes how to instrument Kafka-based applications with distributed tracing capabilities in order to make dataflows between event-based components more visible.

Kafka 111

Kafka Transportation Metadata Consulting 111

Snowflake

NOVEMBER 11, 2024

Unify transactional and analytical workloads in Snowflake for greater simplicity Many businesses must maintain two separate databases: one to handle transactional workloads and another for analytical workloads.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Confluent

APRIL 11, 2019

Unless a use case actively requires a specific database, companies use S3 for storage and process the data with Amazon Elastic MapReduce (EMR) or Amazon Athena. But even if a use case requires a specific database such as Amazon Redshift, data will still land to S3 first and only then load to Redshift. So, it happened.

Kafka 110

Kafka AWS Metadata Architecture 110

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. Microservices. Martin Fowler. This can be answered in two parts: 1.

Kafka 109

Kafka Designing Architecture Coding 109

Pinterest Engineering

NOVEMBER 22, 2023

Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. From these kafka topics, an ingestion service would consume the data points and push them into the GokuS cluster(s) with a retry mechanism (via a separate kafka + small ingestion service) to handle failure.

Database 111

Database Bytes Kafka Architecture 111

Confluent

FEBRUARY 23, 2023

How to use Kafka Streams to aggregate change data capture (CDC) messages from a relational database into transactional messages, powering a scalable microservices architecture.

Kafka 110

Kafka Relational Database Architecture Database 110

DoorDash Engineering

MARCH 27, 2024

At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka. While we have achieved this in databases, it also needs to be extended to other infrastructure components.

Kafka 103

Kafka Architecture Algorithm Process 103

Confluent

NOVEMBER 21, 2019

ksqlDB is a new kind of database purpose-built for stream processing apps, allowing users to build stream processing applications against data in Apache Kafka® and enhancing developer productivity. ksqlDB simplifies […].

Kafka 108

Kafka Database Process Building 108

Confluent

JULY 29, 2020

Change Data Capture (CDC) is an excellent way to introduce streaming analytics into your existing database, and using Debezium enables you to send your change data through Apache Kafka®. Although […].

Kafka 105

Kafka Database MongoDB Data 105

Hevo

APRIL 24, 2025

Based on a report, Apache Kafka stores and streams more than 7 trillion real-time messages per day. To eradicate such complexities, you can use database connecting tools like Debezium and Kafka […]

Kafka 40

Kafka Database Coding Process 40

Confluent

FEBRUARY 16, 2023

description: With real-time data ingestion, streaming, and storage capabilities, Apache Kafka can be used as a database with ksqlDB.

Kafka 72

Kafka Database Data Ingestion Data 72

Confluent

AUGUST 1, 2019

The Kafka Streams API boasts a number of capabilities that make it well suited for maintaining the global state of a distributed system. At Imperva, we took advantage of Kafka Streams to build shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in our system.

Kafka 20

Kafka Systems Building Metadata 20

Data Engineering Podcast

MARCH 2, 2020

The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing. Developers can work with the SQL constructs that they are familiar with while automatically getting the durability and reliability that Kafka offers. How is ksqlDB architected?

Kafka 100

Kafka Process PostgreSQL MySQL 100

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Although the concept of managed services is fairly common in databases (e.g.,

Kafka 21

Kafka Management Cloud AWS 21

Data Engineering Podcast

JANUARY 5, 2020

Summary Databases are useful for inspecting the current state of your application, but inspecting the history of that data can get messy without a way to track changes as they happen. How has the tight coupling with Kafka impacted the direction and capabilities of Debezium? What, if any, other substrates does Debezium support (e.g.

Database 100

Database Kafka PostgreSQL MySQL 100

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. Traditional databases are ill-suited for storing events in high throughput event streams. This document serves as a tutorial and offers best practices for effectively utilizing Faust.

Kafka 98

Kafka Python Process Google Cloud 98

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. However, Apache Kafka is more than just messaging. Some Kafka and Rockset users have also built real-time e-commerce applications , for example, using Rockset’s Java, Node.js

Kafka 21

Kafka SQL BI Hadoop 21

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Confluent

MAY 18, 2020

What is stopping you from using Kafka Streams as your data layer for building applications? After all, it comes with fast, embedded RocksDB storage, takes care of redundancy for you, […].

Kafka 111

Kafka Building IT Database 111

Towards Data Science

FEBRUARY 9, 2024

This involves getting data from an API and storing it in a PostgreSQL database. In the second phase, we’ll develop an application that uses a language model to interact with this database. The second article, which will come later, will delve into creating agents using tools like LangChain to communicate with external databases.

Kafka 98

Kafka Data Engineering Data Engineer PostgreSQL 98

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Microservices, Apache Kafka, and Domain-Driven Design (DDD) covers this in more detail. Example: Severstal.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 104

Database Java SQL Data Ingestion 104

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period.

Kafka 64

Kafka Database High Quality Data Cloud 64

Data Engineering Podcast

SEPTEMBER 12, 2021

Summary The Cassandra database is one of the first open source options for globally scalable storage systems. The community recently released a new major version that marks a milestone in its maturity and stability as a project and database. Since its introduction in 2008 it has been powering systems at every scale.

Database 100

Database Kafka Metadata Data Storage 100

Data Engineering Podcast

JUNE 1, 2020

What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert. What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

MAY 8, 2022

TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. How has the ecosystem of graph databases changed in usage and design in recent years? Start trusting your data with Monte Carlo today! Visit [link] to learn more.

Database 100

Database Data Lake BI Business Intelligence 100

Christophe Blefari

MARCH 15, 2024

Postgres creator launches DBOS, a transactional serverless computing platform — Mike sees DBOS like a cloud-native OS that runs on-top of the database in order to rethink application development and deployment. Unlocking Kafka's potential: tackling tail latency with eBPF.

Metadata 272

Metadata Data Software Engineering Software Engineer 272

Data Engineering Podcast

AUGUST 6, 2022

For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Precisely

MARCH 28, 2025

Some departments used IBM Db2, while others relied on VSAM files or IMS databases creating complex data governance processes and costly data pipeline maintenance. With near real-time data synchronization, the solution ensures that databases stay in sync for reporting, analytics, and data warehousing.

Data Integration 68

Data Integration Pipeline-centric Database-centric Kafka 68

Cloudera

SEPTEMBER 26, 2023

The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog. Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

Kafka 95

Kafka Technology IT Government 95