Cloudera

AUGUST 10, 2022

Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. In CSP, Kafka serves as the storage streaming substrate, and Flink as the core in-stream processing engine that supports SQL and REST interfaces. Apache Kafka and SMM.

Process 90

Process Kafka PostgreSQL MySQL 90

RudderStack

MAY 12, 2021

Finally RudderStack keys "Why they did not prefer Apache Kafka over PostgreSQL for building RudderStack?". Focuses on the challenges using Apache Kafka

Kafka 40

Kafka PostgreSQL Engineering Building 40

Christophe Blefari

JULY 8, 2023

Read dbt metrics documentation As an extension I've seen 2 things this week that I feel makes sense here: VulcanSQL — A data API framework for DuckDB, Snowflake, BigQuery, PostgreSQL. The best way to describe it is: this is a Kafka alternative. Redpanda raises $100m in Series C. Redpanda is a great product for developers.

Kafka 130

Kafka PostgreSQL Data Transportation 130

Data Engineering Weekly

FEBRUARY 4, 2024

The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. It opens some old memory; try to solve this problem first with Presto-Kafka connector and then using OLAP engines like Druid & Apache Pinot. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Confluent

FEBRUARY 12, 2019

One of the most common integrations that people want to do with Apache Kafka ® is getting data in from a database. The existing data in a database, and any changes to that data, can be streamed into a Kafka topic. Here, I’m going to dig into one of the options available—the JDBC connector for Kafka Connect. Introduction.

Kafka 90

Kafka MySQL Bytes Java 90

Snowflake

JUNE 13, 2024

Snowflake is launching native integrations with some of the most popular databases, including PostgreSQL and MySQL. You soon will be able to try out the Snowflake Connectors for PostgreSQL or MySQL by installing them from Snowflake Marketplace and downloading the agent from Docker Hub.

Data Ingestion 90

Data Ingestion MySQL PostgreSQL Data Pipeline 90

Rockset

MARCH 26, 2023

Folks have definitely tried, and while Apache Kafka® has become the standard for event-driven architectures, it still struggles to replace your everyday PostgreSQL database instance in the modern application stack. PostgreSQL, MySQL, SQL Server, and even Oracle are popular choices, but there are many others that will work fine.

Cloud 52

Cloud PostgreSQL Kafka Database 52

Data Engineering Podcast

OCTOBER 15, 2023

To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers Links Decodable Podcast Episode Flink Podcast Episode Debezium Podcast Episode Kafka Redpanda Podcast Episode Kinesis PostgreSQL Podcast Episode Snowflake Podcast Episode Databricks Startree Pinot Podcast Episode Rockset Podcast Episode Druid (..)

Process 182

Process Building SQL BI 182

Data Engineering Podcast

DECEMBER 23, 2018

Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Usman Masood and Derek Nelson about PipelineDB, an open source continuous query engine for PostgreSQL Interview Introduction How did you get involved in the area of data management?

PostgreSQL 100

PostgreSQL Kafka Data Engineering Data Engineer 100

Data Engineering Weekly

OCTOBER 13, 2024

[link] Expedia: Enhancing Data Reliability With An SLO Platform Expedia Group Technology designed a new SLO platform to enhance data reliability, leveraging Kafka for event streaming, PostgreSQL for data storage, and APIs for querying.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Data Engineering Podcast

APRIL 19, 2020

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor.

Building 100

Building PostgreSQL MongoDB SQL 100

Data Engineering Podcast

JANUARY 13, 2019

How have the improvements and new features in the recent releases of PostgreSQL impacted the Timescale product? How have the improvements and new features in the recent releases of PostgreSQL impacted the Timescale product? Have you been able to leverage some of the native improvements to simplify your implementation?

Database 100

Database PostgreSQL SQL MongoDB 100

Monte Carlo

DECEMBER 4, 2023

Tune the load process I'm using PostgreSQL to store my company's transactional data. Implement Upserts I'm frequently updating and inserting new records into my PostgreSQL users table. Provide guidance and best practices on specific ETL tools Say you’re new to Apache Kafka. I've heard about the UPSERT functionality.

PostgreSQL 52

PostgreSQL ETL Tools Data Lake MySQL 52

Data Engineering Podcast

FEBRUARY 11, 2018

release of PostGreSQL had on the design of the project? release of PostGreSQL had on the design of the project? Can you start by explaining what Timescale is and how the project got started? The landscape of time series databases is extensive and oftentimes difficult to navigate. What impact has the 10.0 What impact has the 10.0

PostgreSQL 100

PostgreSQL NoSQL Google Cloud MongoDB 100

Christophe Blefari

JUNE 16, 2023

Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). — Marie wrote best practices for establishing complete and reliable data documentation. The post explains well the architecture you need to make it work.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Knowledge Hut

NOVEMBER 19, 2023

Open Source Support: Many Azure services support popular open-source frameworks like Apache Spark, Kafka, and Hadoop, providing flexibility for data engineering tasks. The Single Server option has been the most often used method of deploying PostgreSQL on the Azure platform up to this point.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Data Engineering Podcast

MAY 27, 2021

Links Yellowbrick Teradata Rainstor Distributed Cloud Hybrid Cloud SwimOS Podcast Episode Kafka Pulsar Podcast Episode Snowflake Podcast Episode AWS Redshift MPP == Massively Parallel Processing Presto Trino Podcast Episode L3 Cache NVMe Reactive Programming Coroutine Star Schema Denodo Lexis Nexis Vertica Netezza Grenplum PostgreSQL Podcast Episode (..)

Data Warehouse 100

Data Warehouse Cloud PostgreSQL Kafka 100

Data Engineering Podcast

JANUARY 5, 2020

How has the tight coupling with Kafka impacted the direction and capabilities of Debezium? How has the tight coupling with Kafka impacted the direction and capabilities of Debezium? What are some of the other options on the market for handling change data capture? What, if any, other substrates does Debezium support (e.g.

Database 100

Database Kafka PostgreSQL MySQL 100

DoorDash Engineering

JANUARY 16, 2024

Storage traffic: Includes traffic from microservices to stateful systems such as Aurora PostgreSQL, CockroachDB, Redis, and Kafka. Explore optimized routing for large data transfer operations to or from DoorDash’s stateful systems, for example PostgreSQL, CRDB, Redis, and Kafka.

Bytes 84

Bytes Cloud Management PostgreSQL 84

LinkedIn Engineering

MAY 11, 2023

There was reliance on an unmanaged data layer.Redis (for caching) and PostgreSQL (as primary datastore)served as single points of failure for this product. Managing data replication for data in PostgreSQL could have been more robust. We decided to leverage Kafka as a distributed messaging queue.

Management 101

Management PostgreSQL MySQL Kafka 101

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. Logstash offers a JDBC input plugin that polls a relational database, like PostgreSQL or MySQL, for inserts and updates periodically.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Data Engineering Podcast

MAY 10, 2021

Links Tinybird Carto PostgreSQL Podcast Episode PostGIS Clickhouse Podcast Episode Kafka Tornado Podcast.__init__ Links Tinybird Carto PostgreSQL Podcast Episode PostGIS Clickhouse Podcast Episode Kafka Tornado Podcast.__init__

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Cloudera

MARCH 23, 2023

An MV is a special type of sink that allows us to output data from our query into a tabular format persisted in a PostgreSQL database. A sink could be another data stream or we could use a special type of data sink we call a materialized view (MV). We can also query this data later, optionally with filters using SSBs REST API.

SQL 81

SQL Kafka PostgreSQL Database 81

Data Engineering Podcast

APRIL 1, 2018

I understand that your original architecture used RabbitMQ as your ingest mechanism, which you then migrated to Kafka. I understand that your original architecture used RabbitMQ as your ingest mechanism, which you then migrated to Kafka. What was your initial motivation for that change? What was your initial motivation for that change?

Amazon Web Services 100

Amazon Web Services Cloud PostgreSQL Kafka 100

Data Engineering Podcast

MAY 20, 2018

Links Starburst Data Presto Hadapt Hadoop Hive Teradata PrestoCare Cost Based Optimizer ANSI SQL Spill To Disk Tempto Benchto Geospatial Functions Cassandra Accumulo Kafka Redis PostGreSQL The intro and outro music is from The Hug by The Freak Fandango Orchestra / {CC BY-SA]([link] Support Data Engineering Podcast

PostgreSQL 100

PostgreSQL Hadoop SQL Kafka 100

Data Engineering Podcast

MARCH 15, 2021

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links How DoorDash is Scaling its Data Platform to Delight Customers and Meet our Growing Demand DoorDash Uber Netscape Netflix Change Data Capture Debezium Podcast (..)

Management 100

Management Data Warehouse PostgreSQL Kafka 100

Data Engineering Podcast

AUGUST 6, 2017

Contact Information Email @rywalker on Twitter Links Astronomer Kiss Metrics Segment Marketing tools chart Clickstream HIPAA FERPA PCI Mesos Mesos DC/OS Airflow SSIS Marathon Prometheus Grafana Terraform Kafka Spark ELK Stack React GraphQL PostGreSQL MongoDB Ceph Druid Aries Vault Adapter Pattern Docker Kinesis API Gateway Kong AWS Lambda Flink Redshift (..)

PostgreSQL 100

PostgreSQL MongoDB Data Pipeline Kafka 100

Christophe Blefari

APRIL 28, 2023

AGI ( credits ) Fast News ⚡️ From PostgreSQL to Snowflake: A data migration story — The migration lasted 9 months and included 8 steps. Technologically moving from Airflow batches to Spark running on top of Kafka. This post showcases a lot of it (it uses a XGBoost model).

SQL 100

SQL Food PostgreSQL Data 100

Data Engineering Weekly

FEBRUARY 12, 2023

link] Etsy: Adding Zonal Resiliency to Etsy’s Kafka Cluster Cross-region (Zone) comes with its penalty of cost and latency in Kafka infrastructure. Etsy writes about resiliency engineering for Kafka infrastructure, adding Zonal resilience in Google Cloud. A must-read for data engineering professionals.

Data Engineering 52

Data Engineering Data Engineer Engineering Hadoop 52

Data Engineering Podcast

AUGUST 6, 2022

To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers Links Milvus Zilliz Linux Foundation/AI & Data MySQL PostgreSQL CockroachDB Pilosa Podcast Episode Pinecone Vector DB Podcast Episode Vector Embedding Reverse Image Search Vector Arithmetic Vector Distance SIGMOD Tensor Rotation Matrix (..)

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Confluent

JUNE 18, 2019

Disclaimer: There are nice projects around like PostgreSQL full-text search that might be enough for your use case, and you should certainly consider them. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®.

Architecture 97

Architecture Building Kafka Database-centric 97

Data Engineering Podcast

NOVEMBER 26, 2019

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Sentry Podcast.__init__

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Edureka

FEBRUARY 9, 2023

Apache Kafka: Apache Kafka is a decentralized publish-subscribe messaging system designed to handle data streams with high volume, high throughput, and low latency. Apache Kafka is highly scalable, fault-tolerant, and scalable, and it provides reliable support for large amounts of data.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Netflix Tech

DECEMBER 17, 2019

In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. In order to be supported, a database is required to fulfill a set of features that are commonly available in systems like MySQL, PostgreSQL, MariaDB, and others. For example in PostgreSQL RDS, changes can only be captured from the master.

MySQL 87

MySQL PostgreSQL Database Transportation 87

Netflix Tech

DECEMBER 17, 2019

In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. In order to be supported, a database is required to fulfill a set of features that are commonly available in systems like MySQL, PostgreSQL, MariaDB, and others. For example in PostgreSQL RDS, changes can only be captured from the master.

MySQL 81

MySQL PostgreSQL Database Transportation 81