Confluent

OCTOBER 16, 2019

Using this data, Apache Kafka ® and Confluent Platform can provide the foundations for both event-driven applications as well as an analytical platform. With tools like KSQL and Kafka Connect, the concept of streaming ETL is made accessible to a much wider audience of developers and data engineers. Ingesting the data.

Kafka 19

Kafka Building Data Coding 19

Confluent

FEBRUARY 12, 2019

One of the most common integrations that people want to do with Apache Kafka ® is getting data in from a database. The existing data in a database, and any changes to that data, can be streamed into a Kafka topic. Here, I’m going to dig into one of the options available—the JDBC connector for Kafka Connect. Introduction.

Kafka 89

Kafka MySQL Bytes Java 89

Cloudera

MARCH 15, 2023

The record in the “outbox” table contains information about the event that happened inside the application, as well as some metadata that is required for further processing or routing. InventoryService) or processing a payment (eg. After the transaction commits, the record will be available for external consumers. PaymentService).

PostgreSQL 91

PostgreSQL Kafka Database Data 91

Data Engineering Podcast

DECEMBER 23, 2018

Summary Processing high velocity time-series data in real-time is a complex challenge. Given the fact that it is a plugin for PostgreSQL, what level of compatibility exists between PipelineDB and other plugins such as Timescale and Citus? Can you start by explaining what PipelineDB is and the motivation for creating it?

PostgreSQL 100

PostgreSQL Kafka Data Engineering Data Engineer 100

Data Engineering Podcast

FEBRUARY 11, 2018

release of PostGreSQL had on the design of the project? release of PostGreSQL had on the design of the project? Can you start by explaining what Timescale is and how the project got started? The landscape of time series databases is extensive and oftentimes difficult to navigate. What impact has the 10.0 What impact has the 10.0

PostgreSQL 100

PostgreSQL NoSQL Google Cloud MongoDB 100

Snowflake

JUNE 13, 2024

Snowflake is launching native integrations with some of the most popular databases, including PostgreSQL and MySQL. With other ingestion improvements and our new database connectors, we are smoothing out the data ingestion process, making it radically simple and efficient to bring data to Snowflake.

Data Ingestion 113

Data Ingestion MySQL PostgreSQL Data Pipeline 113

Data Engineering Podcast

JANUARY 13, 2019

release, how the use cases for timeseries data have proliferated, and how they are continuing to simplify the task of processing your time oriented events. How have the improvements and new features in the recent releases of PostgreSQL impacted the Timescale product?

Database 100

Database PostgreSQL SQL MongoDB 100

Data Engineering Podcast

AUGUST 6, 2017

Astronomer is a platform that lets you skip straight to processing your valuable business data. Regulatory challenges of processing other people’s data What does your data pipelining architecture look like? Astronomer is a platform that lets you skip straight to processing your valuable business data.

PostgreSQL 100

PostgreSQL MongoDB Data Pipeline Kafka 100

Data Engineering Podcast

APRIL 1, 2018

In this episode ThreatStack’s director of operations, Pete Cheslock, and senior infrastructure security engineer, Patrick Cable, discuss the data infrastructure that supports their platform, how they capture and process the data from client systems, and how that information can be used to keep your systems safe from attackers.

Amazon Web Services 100

Amazon Web Services Cloud PostgreSQL Kafka 100

Data Engineering Podcast

APRIL 19, 2020

This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users. What are some of the most interesting, unexpected, or challenging lessons that you have learned in the process of building and scaling Eventador?

Building 100

Building PostgreSQL MongoDB SQL 100

Confluent

JUNE 18, 2019

Disclaimer: There are nice projects around like PostgreSQL full-text search that might be enough for your use case, and you should certainly consider them. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®. Scaling indexing.

Architecture 97

Architecture Building Kafka Database-centric 97

Data Engineering Podcast

MAY 27, 2021

For someone who is adopting Yellowbrick, what is the process for getting it integrated into their data systems? For someone who is adopting Yellowbrick, what is the process for getting it integrated into their data systems? What are some data modeling strategies that users should consider when designing their deployment of Yellowbrick?

Data Warehouse 100

Data Warehouse Cloud PostgreSQL Kafka 100

Data Engineering Podcast

NOVEMBER 26, 2019

In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse. What did the previous system look like? What was your design criteria for building a new platform?

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Netflix Tech

DECEMBER 17, 2019

In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. This motivated the development of DBLog , which offers log and dump processing under a generic framework. Some of DBLog’s features are: Processes captured log events in-order. This way log processing can progress alongside dump processing.

MySQL 89

MySQL PostgreSQL Database Transportation 89

Data Engineering Podcast

OCTOBER 9, 2018

How much processing overhead is involved in the conversion from the column oriented data stored on disk to the row oriented data stored in memory? How much processing overhead is involved in the conversion from the column oriented data stored on disk to the row oriented data stored in memory?

PostgreSQL 100

PostgreSQL BI Machine Learning Data Warehouse 100

LinkedIn Engineering

MAY 11, 2023

With the overarching theme of enabling Site Reliability engineers (SREs) to take ownership of this entire process, we had to think outside the existing solution, which led to designing a tool that could allow direct access to SREs for managing server lifecycle. Managing data replication for data in PostgreSQL could have been more robust.

Management 101

Management PostgreSQL MySQL Kafka 101

Netflix Tech

DECEMBER 17, 2019

In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. This motivated the development of DBLog , which offers log and dump processing under a generic framework. Some of DBLog’s features are: Processes captured log events in-order. This way log processing can progress alongside dump processing.

MySQL 83

MySQL PostgreSQL Database Transportation 83

Data Engineering Podcast

MARCH 15, 2021

Can you give an overview of the collection process for that data? Can you give an overview of the collection process for that data? Can you describe the type(s) of data that you are working with? What are the primary sources of data that you collect? What secondary or third party sources of information do you rely on?

Management 100

Management Data Warehouse PostgreSQL Kafka 100

Data Engineering Podcast

OCTOBER 14, 2019

Your host is Tobias Macey and today I’m interviewing Lewis Hemens about DataForm, a platform that helps analysts manage all data processes in your cloud data warehouse Interview Introduction How did you get involved in the area of data management? Can you talk through some of the use cases that having an embedded runtime enables?

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Data Engineering Podcast

AUGUST 6, 2022

For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. For analytical systems there are decades of investment in data warehouses and various modeling techniques.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Weekly

FEBRUARY 4, 2024

The user journey, sales process, marketing campaign, everything falls under a state machine. Data modeling is a collaborative process across business units to capture state changes in business activity. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Picnic Engineering

MARCH 28, 2025

We started with PostgreSQL, laying the foundation for structured analytics early on. On one hand, the growing volume of data significantly increased processing times, making it difficult to refresh the Data Warehouse overnight. The Turning Point: Year3 At Picnic we had a Data Warehouse from the start, from the very first order.

Data Warehouse 40

Data Warehouse PostgreSQL Python Machine Learning 40

Cloudera

MARCH 23, 2023

Cloudera SQL Stream Builder (SSB) gives the power of a unified stream processing engine to non-technical users so they can integrate, aggregate, query, and analyze both streaming and batch data sources in a single SQL interface. Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). What is a materialized view?

SQL 81

SQL Kafka PostgreSQL Database 81

Zalando Engineering

OCTOBER 9, 2017

These consumers would subscribe, receive, and process the data appropriately for their own needs - essentially inverting the flow of data, from the traditional “pull” based architectures, to a “push” based approach. With this selection of Kafka as the outbound event platform, it was also a natural selection for the inbound data processing.

Kafka 40

Kafka PostgreSQL Database Architecture 40

Rockset

MARCH 26, 2023

Folks have definitely tried, and while Apache Kafka® has become the standard for event-driven architectures, it still struggles to replace your everyday PostgreSQL database instance in the modern application stack. PostgreSQL, MySQL, SQL Server, and even Oracle are popular choices, but there are many others that will work fine.

Cloud 52

Cloud PostgreSQL Kafka Database 52

DoorDash Engineering

JANUARY 16, 2024

Storage traffic: Includes traffic from microservices to stateful systems such as Aurora PostgreSQL, CockroachDB, Redis, and Kafka. This allowed us to enable direct pod-to-pod communication for Iguazu traffic, enabling zone-aware routing while simultaneously reducing the volume of traffic processed by the ELBs as shown in Figure 10.

Bytes 84

Bytes Cloud Management PostgreSQL 84

Snowflake

JUNE 4, 2024

Supporting open storage architectures The AI Data Cloud is a single platform for processing and collaborating on data in a variety of formats, structures and storage locations, including data stored in open file and table formats. Getting data ingested now only takes a few clicks, and the data is encrypted.

Government 108

Government Data Ingestion Data PostgreSQL 108

Data Engineering Weekly

DECEMBER 16, 2024

Vector Search and Unstructured Data Processing Advancements in Search Architecture In 2024, organizations redefined search technology by adopting hybrid architectures that combine traditional keyword-based methods with advanced vector-based approaches.

Data Engineering 62

Data Engineering Data Engineer Engineering Unstructured Data 62

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. Logstash is an event processing pipeline that ingests and transforms data before sending it to Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Zalando Engineering

SEPTEMBER 20, 2015

As part of our learning process, we recently designed and built Saiki : a scalable, cloud-based data integration infrastructure that makes data from our many microservices readily available for analytical teams. This approach allows for a non-intrusive and reliable Change Data Capture of PostgreSQL databases.

Data Integration 40

Data Integration PostgreSQL Amazon Web Services Kafka 40

Monte Carlo

DECEMBER 4, 2023

At the heart of data engineering lies the ETL process—a necessary, if sometimes tedious, set of operations to move data across pipelines for production. Now imagine having a co-pilot to streamline and supercharge the process. Tune the load process I'm using PostgreSQL to store my company's transactional data.

PostgreSQL 52

PostgreSQL Data Lake ETL Tools MySQL 52

Knowledge Hut

NOVEMBER 19, 2023

These tools help in various stages of data processing, storage, and analysis. Open Source Support: Many Azure services support popular open-source frameworks like Apache Spark, Kafka, and Hadoop, providing flexibility for data engineering tasks. Let’s read about them in the next section.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Cloudera

SEPTEMBER 30, 2021

The overall upgrade follows a seven-step process illustrated below. PostgreSQL 10, 11 and 12 and OracleDB 12c, 19c and 19.9. Add Kafka Service – Required for Atlas if it’s not already installed. and will later be converted to Ranger policies and automatically imported during the Upgrade Wizard process.

Cloud 73

Cloud PostgreSQL Metadata MySQL 73

Striim

FEBRUARY 14, 2023

Yet the “Modern Data Stack” is largely focussed on delivering batch processing and reporting on historical data with cloud-native platforms. You see real-time stock tickers on TV, you use real-time odometers when you’re driving to gauge your speed, when you check the weather in your app.

PostgreSQL 52

PostgreSQL MongoDB MySQL Kafka 52

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Rockset

JUNE 16, 2020

An omni-channel retail personalization application, as an example, may require order data from MongoDB, user activity streams from Kafka, and third-party data from a data lake. We can load new data from other data sources—Kafka and Amazon S3—into our production MongoDB instance and run our queries there.

MongoDB 52

MongoDB Data Lake PostgreSQL Kafka 52