Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. How does Postgres reliably ensure that all data change events are emitted and captured by an interested client?

PostgreSQL 138

PostgreSQL Java Database Bytes 138

Data Engineering Podcast

MAY 18, 2020

Summary The PostgreSQL database is massively popular due to its flexibility and extensive ecosystem of extensions, but it is still not the first choice for high performance analytics. If you are trying to get more performance out of your database then this episode is for you! Can you start by explaining what Swarm64 is?

PostgreSQL 100

PostgreSQL Database Data Warehouse Machine Learning 100

Start Data Engineering

JULY 18, 2020

Given Source database connection details (endpoint, port, username, password and database name) Source table to replicate destination schema name run frequency can be set to 10min We are assuming the destination data warehouse is already setup in stitch. Choose PostgreSQL option as the integration in the next page.

Data Warehouse 100

Data Warehouse Database PostgreSQL Data 100

Data Engineering Podcast

JANUARY 7, 2018

Summary PostGreSQL has become one of the most popular and widely used databases, and for good reason. At Citus Data they have built an extension to support running it in a distributed fashion across large volumes of data with parallelized queries for improved performance.

PostgreSQL 100

PostgreSQL Big Data NoSQL Data 100

Data Engineering Podcast

AUGUST 5, 2018

Summary One of the longest running and most popular open source database projects is PostgreSQL. Because of its extensibility and a community focus on stability it has stayed relevant as the ecosystem of development environments and data requirements have changed and evolved over its lifetime.

PostgreSQL 100

PostgreSQL MySQL Banking Database 100

Rock the JVM

MAY 22, 2024

We’ll also need to insert messages into our database, for this we’ll need another case class, create a message.scala file under domain , and add the following code: package rockthejvm.websockets.domain import java.util.UUID import java.time.LocalDateTime import rockthejvm.websockets.domain.user. postgres_volume:/var/lib/postresql/data -./setup.sql:/docker-entrypoint-initdb.d/setup.sql

PostgreSQL 136

PostgreSQL Scala Database SQL 136

Start Data Engineering

JANUARY 1, 2021

Introduction Why Change Data Capture Setup Prerequisites Source setup Destination setup Source, MySQL CDC, MySQL => PostgreSQL Pros and Cons Pros Cons Conclusion References Introduction Change data capture is a software design pattern used to track every change(update, insert, delete) to the data in a database.

PostgreSQL 130

PostgreSQL MySQL Database Data 130

Data Engineering Podcast

JANUARY 5, 2020

Summary Databases are useful for inspecting the current state of your application, but inspecting the history of that data can get messy without a way to track changes as they happen. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council.

Database 100

Database Kafka PostgreSQL MySQL 100

Data Engineering Podcast

APRIL 28, 2019

However, managing the database layer is still a separate concern. In this episode Tamal Saha explains how the KubeDB project got started, why you might want to run your database with Kubernetes, and how to get started. Not only does it free up your engineers’ time, it lets your business users decide what data they want where.

Database 100

Database PostgreSQL MongoDB MySQL 100

Workfall

MARCH 28, 2023

This blog will demonstrate to you how Hasura and PostgreSQL can help you accelerate app development and easily launch backends. In this blog, we will cover: GraphQL Hasura PostgreSQL Hands-on Conclusion GraphQL GraphQL is an API query language and runtime for answering queries with existing data. Why Hasura is Fast?

PostgreSQL 75

PostgreSQL Coding MySQL Database 75

Data Engineering Podcast

JANUARY 13, 2019

release, how the use cases for timeseries data have proliferated, and how they are continuing to simplify the task of processing your time oriented events. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform.

Database 100

Database PostgreSQL SQL MongoDB 100

Data Engineering Podcast

FEBRUARY 11, 2023

Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the business members who play a crucial role in the success of any data project.

PostgreSQL 130

PostgreSQL SQL Machine Learning Data Warehouse 130

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. We spend countless hours designing the data model, updating engine versions, and tuning performance. RudderStack’s smart customer data pipeline is warehouse-first.

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Data Engineering Podcast

JANUARY 22, 2023

Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time consuming and error-prone. Can you describe what Tonic is and the story behind it?

PostgreSQL 130

PostgreSQL MongoDB MySQL Machine Learning 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Materialize]([link] Looking for the simplest way to get the freshest data possible to your teams?

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

MAY 29, 2022

Summary A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required.

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

Confluent

NOVEMBER 19, 2024

CDC has evolved to become a key component of data streaming platforms, and is easily enabled by managed connectors such as the Debezium PostgreSQL CDC connector.

PostgreSQL 64

PostgreSQL Database Data Management 64

Cloudera

DECEMBER 2, 2021

For Cloudera ensuring data security is critical because we have large customers in highly regulated industries like financial services and healthcare, where security is paramount. At Cloudera we want to help all customers to spend more time analyzing data than protecting data. Network Traffic with the CDP Control Plane.

PostgreSQL 128

PostgreSQL Data Lake Data Warehouse Retail 128

Data Engineering Podcast

AUGUST 19, 2018

Summary The way that you store your data can have a huge impact on the ways that it can be practically used. In addition he talks about the challenges of building a distributed, consistent database and the tradeoffs that were made to make DGraph a reality. However, it can be tough learning it when you’re just starting out.

Database 100

Database PostgreSQL NoSQL Transportation 100

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

JANUARY 15, 2023

Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery.

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

JANUARY 8, 2023

Summary Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration of data living in disparate systems. Missing data? Start trusting your data with Monte Carlo today!

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Data Engineering Podcast

OCTOBER 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Process 182

Process Building SQL BI 182

Christophe Blefari

JUNE 24, 2023

( credits ) Hey, this is the Data News. DuckDB is an in-memory analytics database. In a nutshell the announcement is: you can connect to remote DuckDB by doing md: in the connection string and you can join local and remote data (also seen on Twitter ). Doris is a real time analytical database. So it's single server.

PostgreSQL 130

PostgreSQL Data Data Engineering Data Engineer 130

Data Engineering Podcast

JANUARY 29, 2023

Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems.

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

DoorDash Engineering

FEBRUARY 7, 2023

This increased volume of items created high latency and high failure rate in the fulfillment backend mainly caused by database scalability problems. It’s hosted in the PostgreSQL and used to serve item metadata to the Dasher, our name for delivery drivers, during order fulfillment.

PostgreSQL 119

PostgreSQL SQL Retail Database 119

DoorDash Engineering

MARCH 7, 2023

We knew we’d be deploying a Docker container to Fargate as well as using an Amazon Aurora PostgreSQL database and Terraform to model our infrastructure as code. Set up a locally running containerized PostgreSQL database. No chance of accidentally touching any production data or systems.

AWS 119

AWS PostgreSQL Database SQL 119

Christophe Blefari

FEBRUARY 23, 2024

My ideas these days ( credits ) Hey, fresh Data News edition. This week I've participated to a round table about data and did a cool presentation about Engines. — Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Is the modern data stack disappearing?

Data Lake 130

Data Lake PostgreSQL MongoDB MySQL 130

Netflix Tech

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events.

MySQL 89

MySQL PostgreSQL Database Transportation 89

Meltano

OCTOBER 19, 2022

PostgreSQL and MySQL are among the most popular open-source relational database management systems (RDMS) worldwide. Both RDMS enable businesses to organize and interlink large amounts of data, allowing for effective data management. That’s because MySQL isn’t fully SQL-compliant, while PostgreSQL is.

PostgreSQL 52

PostgreSQL MySQL Database SQL 52

Confluent

AUGUST 11, 2020

Building a Cloud ETL Pipeline on Confluent Cloud shows you how to build and deploy a data pipeline entirely in the cloud. However, not all databases can be in the […].

Database 103

Database Kafka Data Pipeline Cloud 103

Hevo

DECEMBER 26, 2023

CSVs are used to transfer data across incompatible database programs, such as between open-source databases and proprietary database formats. PostgreSQL is […] Each entry in the records is separated from another by commas. This is the origin of the name CSV (Comma Separated Values).

PostgreSQL 52

PostgreSQL Database Programming Systems 52

Snowflake

JUNE 13, 2024

The journey toward achieving a robust data platform that secures all your data in one place can seem like a daunting one. But at Snowflake, we’re committed to making the first step the easiest — with seamless, cost-effective data ingestion to help bring your workloads into the AI Data Cloud with ease.

Data Ingestion 94

Data Ingestion MySQL PostgreSQL Data Pipeline 94

Hevo

JUNE 19, 2024

Many organizations are drawn to PostgreSQL’s robust features, open-source nature, and cost-effectiveness, and hence they look to migrate their data from their existing database to PostgreSQL. In this guide, we’ll discuss the Oracle to PostgreSQL migration process.

PostgreSQL 52

PostgreSQL Database Process Data 52

Hevo

DECEMBER 26, 2023

PostgreSQL is an open-source Relational Database taking the world by storm, both on the ground and up there in the Cloud. It is one of the most advanced Relational Databases out there offering standard SQL features along with some modern ones like triggers, transaction integrity, etc.

PostgreSQL 52

PostgreSQL Relational Database Database SQL 52

Zalando Engineering

DECEMBER 1, 2021

This blog post explains to you which tools to use to serve geospatial data from a database system (PostgreSQL) to your web browser. All you need is a database server for the data, a web map application for the frontend and a small service in between to transfer user requests. pg_tileserv is such a solution.

PostgreSQL 52

PostgreSQL Metadata Database SQL 52

Christophe Blefari

JULY 26, 2024

Tallinn ( credits ) Dear members, it's Summer Data News, the only news you can consume by the pool, the beach or at the office—if you're not lucky. Joe is a great speaker, he wrote Fundamentals of Data Engineering , which is one of the bibles in data engineering and I can't wait to hear him at Forward Data.

MySQL 130

MySQL Data PostgreSQL Datasets 130