Data Engineering Podcast

MAY 18, 2020

Summary The PostgreSQL database is massively popular due to its flexibility and extensive ecosystem of extensions, but it is still not the first choice for high performance analytics. Swarm64 aims to change that by adding support for advanced hardware capabilities like FPGAs and optimized usage of modern SSDs.

PostgreSQL 100

PostgreSQL Database Data Warehouse Machine Learning 100

ProjectPro

JUNE 6, 2025

This transformation is where data warehousing tools come into play, acting as the refining process for your data. These tools are crucial in modern business intelligence and data-driven decision-making processes. Amazon Redshift Pros High query performance via columnar storage and parallel processing.

Data Storage 58

Data Storage PostgreSQL Data Warehouse AWS 58

KDnuggets

JUNE 10, 2025

By Josep Ferrer , KDnuggets AI Content Specialist on June 10, 2025 in Python Image by Author DuckDB is a fast, in-process analytical database designed for modern data analysis. DuckDB is a free, open-source, in-process OLAP database built for fast, local analytics. Let’s dive in! What Is DuckDB?

Python 111

Python SQL Data Science Machine Learning 111

Data Engineering Podcast

AUGUST 5, 2018

Summary One of the longest running and most popular open source database projects is PostgreSQL. For anyone who hasn’t used it, can you describe what PostgreSQL is? What are some of the common points of confusion for new users of PostGreSQL? How did you get involved in the Postgres project?

PostgreSQL 100

PostgreSQL MySQL Banking Database 100

Snowflake

NOVEMBER 26, 2024

Before it migrated to Snowflake in 2022, WHOOP was using a catalog of tools — Amazon Redshift for SQL queries and BI tooling, Dremio for a data lake, PostgreSQL databases and others — that had ultimately become expensive to manage and difficult to maintain, let alone scale.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Data Lake 105

Rock the JVM

MAY 22, 2024

Docker for Redis and PostgreSQL We’ll be using Docker images for Redis and Postgres. Skunk for PostgreSQL Integration In this section, we’ll implement the protocols necessary for interacting with Postgres in our application using Skunk. object message {. To follow along, you’ll need Docker and Docker Compose installed. value ), m.

PostgreSQL 136

PostgreSQL Scala Database SQL 136

ProjectPro

JUNE 6, 2025

This blog aims to give you an overview of the data analysis process with a real-world business use case. Table of Contents The Motivation Behind Data Analysis Process What is Data Analysis? What is the goal of the analysis phase of the data analysis process? What are the steps in the data analysis process?

Data Analysis 40

Data Analysis Data Analytics Process Insurance 40

Workfall

MARCH 28, 2023

This blog will demonstrate to you how Hasura and PostgreSQL can help you accelerate app development and easily launch backends. In this blog, we will cover: GraphQL Hasura PostgreSQL Hands-on Conclusion GraphQL GraphQL is an API query language and runtime for answering queries with existing data. Why Hasura is Fast?

PostgreSQL 75

PostgreSQL Coding MySQL Database 75

Data Engineering Weekly

MAY 25, 2025

link] SquareSpace: Leveraging Change Data Capture For Database Migrations At Scale Squarespace writes about migrating their business-critical PostgreSQL databases to CockroachDB (CRDB) at scale. Then, a custom Apache Beam consumer processed these events, transforming and writing them to CRDB.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Towards Data Science

DECEMBER 10, 2024

From governance processes to costly tools to dbt implementationdata quality projects never seem to want to besmall. And would you believe all of this was available to us since the release of PostgreSQL 6.5 Set up a development and testing process e.g. development environment, version control,CI/CD. Precise decimal handling.

PostgreSQL 75

PostgreSQL Data SQL Python 75

Cloudera

AUGUST 10, 2022

Cloudera has a strong track record of providing a comprehensive solution for stream processing. Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. Cloudera Stream Processing Community Edition.

Process 91

Process Kafka PostgreSQL MySQL 91

Data Engineering Podcast

MARCH 2, 2020

The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

ProjectPro

JUNE 6, 2025

Microsoft Azure Data Factory Microsoft Azure Data Factory ( ADF ) is a fully-managed, serverless data integration tool for acquiring, analyzing, and processing all of your data in bulk. The store supports low-latency workloads and facilitates high-performance processing and analytics from HDFS applications and tools.

Data Engineering 40

Data Engineering Data Engineer PostgreSQL Engineering 40

ProjectPro

JUNE 6, 2025

Looking for an efficient tool for streamlining and automating your data processing workflows? Let's consider an example of a data processing pipeline that involves ingesting data from various sources, cleaning it, and then performing analysis. Airflow operators hold the data processing logic.

Data Pipeline 63

Data Pipeline PostgreSQL Python Database 63

ProjectPro

JUNE 6, 2025

RDS is a fully-managed service that sets up and manages cloud-based database servers, while Aurora Serverless is a relational database engine with a more advanced deployment process that does not require manual management of database servers. On the other hand, RDS only supports five replicas, and its replication process is slower.

Database 40

Database PostgreSQL MySQL Relational Database 40

Start Data Engineering

SEPTEMBER 26, 2020

Table of Contents Table of Contents Introduction Project description and requirements Infrastructure overview Apache Flink Apache Kafka Design Detect fraudulent accounts Log account actions Prerequisites Code Defining dependencies Inheritance Server logs generator Defining data flow in Apache Flink Create a streaming environment Creating a consumer (..)

Data Engineering 130

Data Engineering Data Engineer Project Engineering 130

ProjectPro

JUNE 6, 2025

Traditional ETL processes have long been a bottleneck for businesses looking to turn raw data into actionable insights. Zero ETL integrations minimize this complexity by automating data mapping and reducing the need for manual intervention, making the entire process more streamlined and efficient. How Does Zero-ETL Work?

Data Analytics 40

Data Analytics MySQL PostgreSQL Data Lake 40

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Optimizing the server initialization process for Atlas is vital for maintaining the high availability and performance of the ThoughtSpot system.

Metadata 52

Metadata PostgreSQL Java Database 52

Hevo

JUNE 19, 2024

Many organizations are drawn to PostgreSQL’s robust features, open-source nature, and cost-effectiveness, and hence they look to migrate their data from their existing database to PostgreSQL. In this guide, we’ll discuss the Oracle to PostgreSQL migration process.

PostgreSQL 52

PostgreSQL Database Process Data 52

ProjectPro

JUNE 6, 2025

Since Amazon Redshift is based on the industry standard PostgreSQL, several SQL client applications work with minimum changes. Using Apache Airflow with Python programming language, you can build a reusable and parameterizable ETL process that will digest data from the S3 bucket into Redshift.

Data Pipeline 64

Data Pipeline AWS Project Building 64

Meltano

OCTOBER 19, 2022

PostgreSQL and MySQL are among the most popular open-source relational database management systems (RDMS) worldwide. For all of their similarities, PostgreSQL and MySQL differ from one another in many ways. That’s because MySQL isn’t fully SQL-compliant, while PostgreSQL is.

PostgreSQL 52

PostgreSQL MySQL Database SQL 52

ProjectPro

JUNE 6, 2025

Conceptual data modeling refers to the process of creating conceptual data models. Physical data modeling is the process of creating physical data models. This is the process of putting a conceptual data model into action and extending it. The process of creating logical data models is known as logical data modeling.

Data Warehouse 53

Data Warehouse NoSQL PostgreSQL Relational Database 53

Knowledge Hut

DECEMBER 21, 2023

In the database ecosystem, Postgres is one of the top open-source databases, and one of the most widely used PSQL tools for managing PostgreSQL is pgAdmin. To run PostgreSQL instances on the Azure cloud, Azure offers Azure Database for PostgreSQL. What are PostgreSQL Tools? Why Use a GUI Tool?

PostgreSQL 52

PostgreSQL MySQL MongoDB SQL 52

ProjectPro

JUNE 6, 2025

You’ll walk through each stage of the data processing workflow, similar to what’s used in production-grade systems. Extract, Transform, and Load (ETL) is a process that lies at the core of every application, from dashboards to machine learning models. You don’t want to do this manually every day, right?

Python 40

Python Building PostgreSQL Raw Data 40

ProjectPro

JUNE 6, 2025

This comprehensive blog will explore the key benefits and features of AWS Aurora and also discuss how Aurora compares to traditional enterprise databases like MySQL and PostgreSQL. Aurora also supports parallel query processing, which can significantly speed up complex queries.

Database 40

Database Technology Cloud PostgreSQL 40

Hevo

AUGUST 15, 2023

MongoDB Atlas excels at storing and processing unstructured and semi-structured data, while PostgreSQL offers scalability and advanced analytics. MongoDB Atlas to PostgreSQL integration forms a robust ecosystem that addresses the technical challenges associated with data management and analysis.

MongoDB 52

MongoDB PostgreSQL Structured Data Data 52

ProjectPro

JUNE 6, 2025

The process of merging and integrating data from several sources into a logical, unified view of data is known as data integration. The process of merging and integrating data from several sources into a logical, unified view of data is known as data integration. Data integration projects revolve around managing this process.

Data Integration 40

Data Integration Project Data Lake PostgreSQL 40

ProjectPro

JUNE 6, 2025

Such an immense volume of data requires more than just storage; it demands complex data processing workloads to organize, manage, and analyze it effectively. They include relational databases like Amazon RDS for MySQL, PostgreSQL, and Oracle and NoSQL databases like Amazon DynamoDB.

AWS 40

AWS Database Amazon Web Services MySQL 40

Data Engineering Podcast

JANUARY 29, 2023

Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Hevo

SEPTEMBER 20, 2023

Amazon RDS, with its support for the PostgreSQL database, is a popular choice for businesses looking for reliable relational database services. However, the increasing need for advanced analytics and large-scale data processing requires migrating data to more efficient platforms like Databricks.

PostgreSQL 52

PostgreSQL Relational Database Database Data 52

Cloudera

DECEMBER 2, 2021

Also, for other industries like retail, telecom or public sector that deal with large amounts of customer data and operate multi-tenant environments, sometimes with end users who are outside of their company, securing all the data may be a very time intensive process. CDW uses various Azure services to provide the infrastructure it requires.

PostgreSQL 123

PostgreSQL Data Lake Data Warehouse Retail 123

DoorDash Engineering

MARCH 7, 2023

We knew we’d be deploying a Docker container to Fargate as well as using an Amazon Aurora PostgreSQL database and Terraform to model our infrastructure as code. Set up a locally running containerized PostgreSQL database. This isn’t necessary for your application, but it definitely speeds up the development process.

AWS 119

AWS PostgreSQL Database SQL 119

ProjectPro

JUNE 6, 2025

Additionally, it natively supports data hosted in Amazon Aurora , Amazon RDS, Amazon Redshift , DynamoDB, and Amazon S3, along with JDBC-type data stores such as MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases in your Amazon Virtual Private Cloud, and MongoDB client stores (MongoDB, Amazon DocumentDB). Libraries No.

ETL Tools 40

ETL Tools AWS Data Pipeline Amazon Web Services 40

Hevo

SEPTEMBER 22, 2023

Google Cloud SQL for PostgreSQL, a part of Google’s robust cloud ecosystem, offers businesses a dependable solution for managing relational data. However, with the expanding need for advanced data analytics, it is required to integrate data storage and processing platforms like Snowflake.

Google Cloud 52

Google Cloud PostgreSQL SQL Cloud 52

Hevo

AUGUST 18, 2023

There are several reasons why data replication from PostgreSQL on Amazon RDS to SQL Server may become necessary. These reasons include changes in business processes, increased data volumes, and enhanced performance requirements.

PostgreSQL 52

PostgreSQL SQL Data Security Data 52

ProjectPro

JUNE 6, 2025

Use Cases for General Purpose RDS Instances The M instance family is ideal for small to medium-sized databases, memory-intensive data processing activities, cluster computing, and other enterprise applications.If Relational databases like MySQL and PostgreSQL. In-memory databases like Redis and Memcached. micro, db.t3.micro,

PostgreSQL 40

PostgreSQL MySQL AWS Relational Database 40

ProjectPro

JUNE 6, 2025

Data Migration Process | What are the Steps Involved in Data Migration? This means that data migration and integration processes must be efficient and seamless, regardless of whether the data is moving from a source to a data lake, from a data warehouse to a data mart, or any other destination system.

PostgreSQL 40

PostgreSQL AWS Data Warehouse Database 40