Confluent

JULY 17, 2019

Together, MongoDB and Apache Kafka ® make up the heart of many modern data architectures today. Integrating Kafka with external systems like MongoDB is best done though the use of Kafka Connect. The official MongoDB Connector for Apache Kafka is developed and supported by MongoDB engineers. Getting started.

MongoDB 21

MongoDB Kafka Database Medical 21

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. Microservices. Martin Fowler. This can be answered in two parts: 1.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Towards Data Science

MARCH 29, 2023

In this article, I want to focus on my on-again, off-again relationship with books and reading. I burned out spectacularly about a year into a PhD program, and my relationship with books ended for quite some time. Even if you haven’t read any of the books below, you’ve probably at least heard of some of them.

Data Warehouse 96

Data Warehouse BI Healthcare Database 96

Confluent

APRIL 30, 2019

With the release of Apache Kafka ® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. In what follows, we provide some context around how a processor topology was generated inside Kafka Streams before 2.1, Kafka Streams topology generation 101.

Kafka 91

Kafka Coding Bytes Process 91

Knowledge Hut

MAY 3, 2024

Dean Wampler (Renowned author of many big data technology-related books) Dean Wampler makes an important point in one of his webinars. Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools.

Kafka 98

Kafka Scala Java Amazon Web Services 98

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop ETL Tools Java 93

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Testing the Kafka Connect Neo4j Sink. curl -X POST [link]. jar -f AVRO -e 100000.

Kafka 88

Kafka Java Programming Language Data Pipeline 88

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. In the first section, I present an introductory overview of stream processing concepts, drawing extensively from the book Designing Data-Intensive Applications [1].

Kafka 98

Kafka Python Process Google Cloud 98

Confluent

OCTOBER 10, 2019

Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets. Microservices, Apache Kafka, and Domain-Driven Design (DDD) covers this in more detail. Example: Severstal.

Kafka 20

Kafka Google Cloud Architecture Java 20

ProjectPro

JUNE 6, 2025

Apache Spark Streaming Use Cases Spark Streaming Architecture: Discretized Streams Spark Streaming Example in Java Spark Streaming vs. Structured Streaming Spark Streaming Structured Streaming What is Kafka Streaming? Kafka Stream vs. Spark Streaming What is Spark streaming? What is Kafka Streaming?

Architecture 45

Architecture Kafka Java Scala 45

ProjectPro

JUNE 6, 2025

Use Kafka for real-time data ingestion, preprocess with Apache Spark, and store data in Snowflake. This architecture shows that simulated sensor data is ingested from MQTT to Kafka. The data in Kafka is analyzed with Spark Streaming API and stored in a column store called HBase.

Data Engineer 59

Data Engineer Data Engineering Project Engineering 59

Data Engineering Podcast

DECEMBER 9, 2018

Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. How does it compare to some of the other streaming frameworks such as Flink, Kafka, or Storm? What was your motivation for writing a book about Spark? Who uses Spark?

Scala 100

Scala Kafka MySQL Hadoop 100

Christophe Blefari

JULY 13, 2024

DevOps for data science — An open-source and free book covering what data scientists need to know about DevOps. Apache Kafka overview — If you're not familiar with Kafka this is a great overview. It has been written by someone at Posit (the company behind RStudio).

Kafka 130

Kafka AWS Data Database 130

Data Engineering Podcast

DECEMBER 10, 2017

When you are processing that data in multiple systems it can be difficult to ensure that they all have an accurate representation of that schema, which is why Confluent has built a schema registry that plugs into Kafka. Conversely, what would be involved in using a storage backend other than Kafka?

Kafka 100

Kafka Data Pipeline Data Engineer Data Engineering 100

The Pragmatic Engineer

JUNE 13, 2023

Agoda is a leading online travel booking platform in Asia. It’s owned by Booking Holdings Inc, which also owns the popular travel sites, Kayak and Booking.com. Unlike Uber, Agoda does not make use of public cloud providers, having decided to build out its own private cloud, instead.

Cloud 253

Cloud Database Utilities BI 253

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka BI Big Data Data Engineer 100

Knowledge Hut

JUNE 30, 2023

Whether you're a beginner looking to dive into the foundations or an experienced practitioner seeking advanced techniques, the right books can be your guiding light. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 18, 2018

How does Flink compare to other streaming engines such as Spark, Kafka, Pulsar, and Storm? How does Flink compare to other streaming engines such as Spark, Kafka, Pulsar, and Storm? Can you start by describing what Flink is and how the project got started? What are some of the primary ways that Flink is used? How is Flink architected?

Process 100

Process Scala Kafka Google Cloud 100

Data Engineering Podcast

JULY 6, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? What have you found to be the most notable evolutions in the community and ecosystem around Kafka and streaming platforms? If you hand a book to a new data engineer, what wisdom would you add to it? Redis and Pulsar)?

Systems 100

Systems Kafka SQL Government 100

Confluent

JULY 11, 2019

Reading, writing, and transforming data in Apache Kafka ® using KSQL is an effective way to rapidly deliver event streaming applications for clients (e.g., For a KSQL newbie the practical exercises show you how to process data in Apache Kafka using an interactive SQL interface. streaming insurance events ).

Kafka 85

Kafka Insurance SQL Architecture 85

Data Engineering Podcast

AUGUST 3, 2021

Now there’s a book that captures the foundational lessons and principles that underly everything that you hear about here. Now there’s a book that captures the foundational lessons and principles that underly everything that you hear about here.

Data Lake 130

Data Lake Data Warehouse Hadoop Kafka 130

Christophe Blefari

JANUARY 14, 2023

How query engines work — This is a web book that explains how query engines work. Analysis of Confluent buying Immerok — Jesse Anderson analyses last week news of Confluent (Kafka) buying Immerok (Flink) and what it implies in the real-time low-level technologies competition between Kafka / Flink / Spark.

Kafka 130

Kafka Data Python Scala 130

Data Engineering Podcast

JULY 27, 2021

Now there’s a book that captures the foundational lessons and principles that underly everything that you hear about here. How do you handle migrating existing projects, particularly if they are using Kafka currently? How do you handle migrating existing projects, particularly if they are using Kafka currently?

Management 100

Management Building Kafka Data Warehouse 100

Christophe Blefari

JANUARY 20, 2024

This book, 📘 Data Pipelines Pocket Reference , defines everything related to data pipelines and how to treat data movement from source to target. Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. workflows (Airflow, Prefect, Dagster, etc.) This is not.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Podcast

DECEMBER 24, 2017

Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today.

Kafka 100

Kafka Python Data Engineer Data Engineering 100

Data Engineering Podcast

OCTOBER 5, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help.

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100

Confluent

FEBRUARY 27, 2019

When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. By persisting the streams in Kafka we then have a record of all system activity (a source of truth), and also a mechanism to drive reactions.

Kafka 109

Kafka AWS Architecture Cloud 109

Monte Carlo

NOVEMBER 21, 2024

For over 20 years, Skyscanner has been helping travelers plan and book trips with confidence— including airfare, hotels, and car rentals. As digital natives, the organization is no stranger to staggering volume.

Government 52

Government Datasets Data Governance Data 52

Monte Carlo

NOVEMBER 21, 2024

For over 20 years, Skyscanner has been helping travelers plan and book trips with confidence— including airfare, hotels, and car rentals. As digital natives, the organization is no stranger to staggering volume.

Government 52

Government Datasets Data Governance Data 52

Data Engineering Podcast

JUNE 15, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help.

Machine Learning 100

Machine Learning Google Cloud Kafka Data Engineer 100

Data Engineering Podcast

JUNE 1, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Data Engineering Podcast

DECEMBER 4, 2022

Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Towards Data Science

DECEMBER 4, 2023

We’ll have a long list of systems to integrate and each will be supporting a different protocol or interface: Kafka Streaming, SFTP, MQTT, REST API and more. Data Architecture — Overview Conclusions Data Engineering is a magical realm, with a plethora of books dedicated to it.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Data Engineering Podcast

AUGUST 24, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help.

Metadata 100

Metadata Management Kafka Data Engineer 100

Data Engineering Podcast

SEPTEMBER 21, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help.

Data Engineer 100

Data Engineer Data Engineering Engineering AWS 100

Data Engineering Podcast

OCTOBER 12, 2020

If you hand a book to a new data engineer, what wisdom would you add to it? Equalum also leverages open source data frameworks by orchestrating Apache Spark, Kafka and others under the hood. If you hand a book to a new data engineer, what wisdom would you add to it?

Business Intelligence 100

Business Intelligence BI Consulting Data Ingestion 100

Data Engineering Podcast

JULY 16, 2021

Now there’s a book that captures the foundational lessons and principles that underly everything that you hear about here. Now there’s a book that captures the foundational lessons and principles that underly everything that you hear about here.

High Quality Data 100

High Quality Data Data Engineer Data Engineering Coding 100