Confluent

FEBRUARY 12, 2019

One of the most common integrations that people want to do with Apache Kafka ® is getting data in from a database. That is because relational databases are a rich source of events. The existing data in a database, and any changes to that data, can be streamed into a Kafka topic. What we’ll cover.

Kafka 90

Kafka MySQL Bytes Java 90

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. gradlew composeUp. The KSQL pipeline flow.

Kafka 96

Kafka Management Bytes SQL 96

Confluent

SEPTEMBER 19, 2019

When people ask me the very top-level question “why do people use Kafka,” I usually lead with the story in my last post , where I talked about how Apache Kafka ® is helping us deliver on the promises the cloud made to us a decade ago. Industry heavyweights like Capital One use event streaming on Kafka for this very task.

Kafka 14

Kafka Data Pipeline Bytes Data Architect 14

Grouparoo

OCTOBER 6, 2021

39 How to Prevent a Data Mutiny Key trends: modular architecture, declarative configuration, automated systems 40 Know the Value per Byte of Your Data Check if you are actually using your data 41 Know Your Latencies key questions: how old is data? 55 Pipe Dreams Kafka was good because it had replaying of messages. Increase visibility.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

ProjectPro

FEBRUARY 21, 2023

Exabytes are 10006 bytes, so to put it into perspective, 463 exabytes is the same as 212,765,957 DVDs. Proficiency in data ingestion, including the ability to import and export data between your cluster and external relational database management systems and ingest real-time and near-real-time (NRT) streaming data into HDFS.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

ProjectPro

NOVEMBER 22, 2021

For input streams receiving data through networks such as Kafka, Flume, and others, the default persistence level setting is configured to achieve data replication on two nodes to achieve fault tolerance. MEMORY ONLY SER: The RDD is stored as One Byte per partition serialized Java Objects.

Hadoop 52

Hadoop Python Datasets Metadata 52

Towards Data Science

JANUARY 7, 2025

Change Data Capture (CDC) is a powerful and efficient tool for transmitting data changes from relational databases such as MySQL and PostgreSQL. PostgreSQL (Physical Replication) : Uses Write-Ahead Logs (WAL), which record low-level changes to the database at a disk block level.

PostgreSQL 84

PostgreSQL MySQL Data Lake Bytes 84