Rockset

APRIL 15, 2021

Rockset replicates the data in real-time from your primary database, including both the initial full-copy data replication into Rockset and staying in sync by continuously reading your MySQL or PostgreSQL change streams.

PostgreSQL 40

PostgreSQL MySQL Relational Database NoSQL 40

Data Engineering Weekly

FEBRUARY 4, 2024

The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. It opens some old memory; try to solve this problem first with Presto-Kafka connector and then using OLAP engines like Druid & Apache Pinot. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Podcast

OCTOBER 15, 2023

NODES Conference Logo]([link] NODES 2023 is a free online conference focused on graph-driven innovations with content for all skill levels. Its 24 hours are packed with 90 interactive technical sessions from top developers and data scientists across the world covering a broad range of topics and use cases.

Process 182

Process Building SQL BI 182

Data Engineering Weekly

OCTOBER 13, 2024

The platform efficiently ingests and enriches data from multiple sources with internal metadata, providing near real-time access and seamless integration with DataDog for proactive monitoring and real-time alerting.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Confluent

FEBRUARY 12, 2019

One of the most common integrations that people want to do with Apache Kafka ® is getting data in from a database. That is because relational databases are a rich source of events. The existing data in a database, and any changes to that data, can be streamed into a Kafka topic. Setting the Kafka message key.

Kafka 90

Kafka MySQL Bytes Java 90

Data Engineering Podcast

DECEMBER 23, 2018

The team at PipelineDB has built a continuous query engine that simplifies the task of computing aggregates across incoming streams of events. Given the fact that it is a plugin for PostgreSQL, what level of compatibility exists between PipelineDB and other plugins such as Timescale and Citus?

PostgreSQL 100

PostgreSQL Kafka Data Engineering Data Engineer 100

Data Engineering Podcast

MAY 10, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. Links Tinybird Carto PostgreSQL Podcast Episode PostGIS Clickhouse Podcast Episode Kafka Tornado Podcast.__init__

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Data Engineering Podcast

JANUARY 13, 2019

release, how the use cases for timeseries data have proliferated, and how they are continuing to simplify the task of processing your time oriented events. How have the improvements and new features in the recent releases of PostgreSQL impacted the Timescale product?

Database 100

Database PostgreSQL SQL MongoDB 100

Data Engineering Podcast

JANUARY 5, 2020

Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. Pulsar, Bookkeeper, Pravega)?

Database 100

Database Kafka PostgreSQL MySQL 100

Knowledge Hut

NOVEMBER 19, 2023

Open Source Support: Many Azure services support popular open-source frameworks like Apache Spark, Kafka, and Hadoop, providing flexibility for data engineering tasks. These patterns can be used to start workflows, trigger events, or create alerts, as well as to feed data into reporting tools or store altered data for later use.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Data Engineering Podcast

MARCH 15, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Management 100

Management Data Warehouse PostgreSQL Kafka 100

Data Engineering Weekly

FEBRUARY 12, 2023

Historical data processing is a rare event, where 99% of the computing happens over the last 24 hours of data. link] Mixpanel: Tracking events at millisecond granularity My first reaction while reading “Historically, Mixpanel used to track events at second-level granularity.”, ”, Wait, what?

Data Engineering 52

Data Engineering Data Engineer Engineering Hadoop 52

Netflix Tech

DECEMBER 17, 2019

In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. In order to be supported, a database is required to fulfill a set of features that are commonly available in systems like MySQL, PostgreSQL, MariaDB, and others. Some of DBLog’s features are: Processes captured log events in-order.

MySQL 87

MySQL PostgreSQL Database Transportation 87

Netflix Tech

DECEMBER 17, 2019

In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. In order to be supported, a database is required to fulfill a set of features that are commonly available in systems like MySQL, PostgreSQL, MariaDB, and others. Some of DBLog’s features are: Processes captured log events in-order.

MySQL 81

MySQL PostgreSQL Database Transportation 81

DoorDash Engineering

JANUARY 16, 2024

Storage traffic: Includes traffic from microservices to stateful systems such as Aurora PostgreSQL, CockroachDB, Redis, and Kafka. Around 20% of this cost was caused by traffic to one elastic load balancer (ELB) used for our event ingestion pipeline ( Iguazu ).

Bytes 84

Bytes Cloud Management PostgreSQL 84

Data Engineering Podcast

AUGUST 6, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. Logstash is an event processing pipeline that ingests and transforms data before sending it to Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Cloudera

MARCH 23, 2023

This allows business users to define events of interest for which they need to continuously monitor and respond quickly. An MV is a special type of sink that allows us to output data from our query into a tabular format persisted in a PostgreSQL database. We can also query this data later, optionally with filters using SSBs REST API.

SQL 80

SQL Kafka PostgreSQL Database 80

LinkedIn Engineering

MAY 11, 2023

We leveraged Iris -based alerting and an internally available event bus through which SREs could know the state of their submissions without continually interacting with the API. There was reliance on an unmanaged data layer.Redis (for caching) and PostgreSQL (as primary datastore)served as single points of failure for this product.

Management 101

Management PostgreSQL MySQL Kafka 101

Data Engineering Podcast

OCTOBER 14, 2019

Upcoming events include the combined events of the Data Architecture Summit and Graphorum, the Data Orchestration Summit, and Data Council in NYC. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Hevo

MAY 23, 2023

Debezium is an open-source, distributed system that can convert real-time changes of existing databases into event streams so that various applications can consume and respond immediately. Debezium uses connectors like PostgreSQL, SQL, MySQL, Oracle, MongoDB, and more for respective databases to stream such changes.

PostgreSQL 52

PostgreSQL MongoDB MySQL SQL 52

DoorDash Engineering

AUGUST 15, 2023

As shown in Figure 1 below, BPFAgent first instruments the kernel via our eBPF probes to capture and produce events. These events then go to processors to be enriched based on process and Kubernetes information. Ultimately, the enriched events are emitted via an exporter to our data store.

Bytes 84

Bytes PostgreSQL Coding Database 84

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Striim

JANUARY 4, 2023

On a technical level, data contracts handle schemas of entities and events. In their example, they use Docker compose to spin up a test instance of their database, a CDC pipeline (using Debezium), Kafka, and the Confluent Schema Registry. Striim is using Streaming SQL to filter tables in your PostgreSQL based on data contracts.

PostgreSQL 52

PostgreSQL Data Warehouse Data Data Lake 52

Data Engineering Podcast

DECEMBER 16, 2019

Upcoming events include the Software Architecture Conference, the Strata Data conference, and PyCon US. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. How is the metadata itself stored and managed in Marquez?

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Confluent

JUNE 18, 2019

Disclaimer: There are nice projects around like PostgreSQL full-text search that might be enough for your use case, and you should certainly consider them. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®.

Architecture 97

Architecture Building Kafka Database-centric 97

Striim

DECEMBER 8, 2023

Change Data Capture to Multiple Subscribers It is common for organizations to stream transactional data from a database to multiple consumers – whether it be different lines of the business or separate technical infrastructure (databases, data warehouses, and messaging systems like Kafka). Source (e.g., toString().equals("QATEST.DEPARTMENTS")

Kafka 64

Kafka Database Data Warehouse Data 64

Rockset

MARCH 10, 2022

To deliver real-time analytics, companies need a modern technology infrastructure that includes these three things: A real-time data source such as web clickstreams, IoT events produced by sensors, etc. A platform such as Apache Kafka/Confluent , Spark or Amazon Kinesis for publishing that stream of event data.

Data Analytics 52

Data Analytics Data Warehouse Medical MySQL 52

Edureka

FEBRUARY 9, 2023

Apache Kafka: Apache Kafka is a decentralized publish-subscribe messaging system designed to handle data streams with high volume, high throughput, and low latency. It is ideal for use cases such as real-time data processing, data streaming, and event-driven architectures because it allows multiple consumers to read data concurrently.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Striim

NOVEMBER 13, 2023

Snowflake Writer: Technical Deep Dive on File-based uploads The SnowflakeWriter in Striim is a robust tool that stages events to local storage, AWS S3, or Azure Storage, then writes to Snowflake according to the defined Upload Policy. Customizable Upload Policy: Allows defining batch uploads based on event count, time intervals, or file size.

Data Ingestion 59

Data Ingestion Utilities Data Integration Data 59

Rockset

AUGUST 30, 2021

Apache Kafka has made acquiring real-time data more mainstream, but only a small sliver are turning batch analytics, run nightly, into real-time analytical dashboards with alerts and automatic anomaly detection. But until this release, all these data sources involved indexing the incoming raw data on a record by record basis.

SQL 52

SQL Kafka MongoDB MySQL 52

Ascend.io

SEPTEMBER 14, 2023

Be it PostgreSQL, MySQL, MongoDB, or Cassandra, Python ensures seamless interactions. Use Case: Storing data with PostgreSQL (example) import psycopg2 conn = psycopg2.connect(dbname="mydb", Use Case: Storing data with PostgreSQL (example) import psycopg2 conn = psycopg2.connect(dbname="mydb",

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Zalando Engineering

SEPTEMBER 20, 2015

Named after the Javanese word for “queue,” Saiki is built mostly in Python and includes components that provide a scalable Change Data Capture infrastructure, consume PostgreSQL replication logs, and perform other relevant tasks. Accessing : Typically a microservice or application has to push event notifications to one of our APIs.

Data Integration 40

Data Integration PostgreSQL Amazon Web Services Kafka 40

Cloudera

APRIL 15, 2021

Added support for standalone NiFi/Kafka clusters. Ranger Audit filters help with using JSON defined filters to control the audit events captured in order to streamline audit volumes by only including relevant events. . Automatic Dynamic Queue support for parent and child queues. We’ve added OS support for RHEL / CentOS 7.9

Cloud 70

Cloud MySQL PostgreSQL SQL 70

Confluent

OCTOBER 24, 2019

I’ve written an event sourcing bank simulation in Clojure (a lisp build for Java virtual machines or JVMs) called open-bank-mark , which you are welcome to read about in my previous blog post explaining the story behind this open source example. Either way, both are accomplished with event sourcing. The bank application.

Kafka 18

Kafka Java Banking Bytes 18

Zalando Engineering

FEBRUARY 6, 2019

Nakadi is a distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues. Project Highlights Lionel Montrieux brought Nakadi to FOSDEM 2019. This is one of the largest open source projects released by Zalando. Try out Nakadi !

PostgreSQL 40

PostgreSQL Kafka Project Technology 40