Christophe Blefari

JANUARY 11, 2025

The future of data querying with Natural Language — What are all the architecture block needed to make natural language query working with data (esp. Hard data integration problems — As always Max describes the best way the reality. when you have a semantic layer).

Data 130

Data Data Warehouse Coding Programming Language 130

Striim

OCTOBER 31, 2024

With Striim’s real-time data integration solution, the institution successfully transitioned to a cloud infrastructure, maintaining seamless operations and paving the way for future advancements. After evaluating various options, they selected Striim for its real-time data integration and streaming capabilities.

Education 52

Education Healthcare Data Integration Cloud 52

Data Engineering Podcast

NOVEMBER 30, 2020

Summary The first stage of every good pipeline is to perform data integration. With the increasing pace of change and the need for up to date analytics the need to integrate that data in near real time is growing. There are a number of projects and platforms on the market that target data integration.

Data Integration 100

Data Integration Coding BI Data Engineer 100

Data Engineering Podcast

AUGUST 27, 2021

Summary The reason that so much time and energy is spent on data integration is because of how our applications are designed. By making the software be the owner of the data that it generates, we have to go through the trouble of extracting the information to then be used elsewhere. What is Zero-Copy Integration?

Data Integration 100

Data Integration Architecture Data Warehouse Data Lake 100

Data Engineering Podcast

OCTOBER 5, 2020

Summary Analytical workloads require a well engineered and well maintained data integration process to ensure that your information is reliable and up to date. Building a real-time pipeline for your data lakes and data warehouses is a non-trivial effort, requiring a substantial investment of time and energy.

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100

Data Engineering Podcast

JANUARY 25, 2021

In order to reduce the friction involved in supporting new data transformations David Molot and Hassan Syyid built the Hotlue platform. Monitor all your databases, cloud services, containers, and serverless functions in one place with Datadog’s 400+ vendor-backed integrations.

Data Integration 100

Data Integration IT Python BI 100

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineer 50

Towards Data Science

OCTOBER 12, 2023

To gather all the necessary information we need to infere a Database Schema to ChatGPT including example datasets and field descriptions by using few-shot prompting. We will start out propagating the Database Schema and some example data to ChatGPT.

Data Integration 97

Data Integration Systems Engineering Datasets 97

Data Engineering Podcast

MAY 1, 2022

Summary The predominant pattern for data integration in the cloud has become extract, load, and then transform or ELT. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Start trusting your data with Monte Carlo today!

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Precisely

FEBRUARY 9, 2024

In 2023, organizations dealt with more data than ever and witnessed a surge in demand for artificial intelligence use cases – particularly driven by generative AI. They relied on their data as a critical factor to guide their businesses to agility and success.

Data Integration 67

Data Integration Government Data Metadata 67

Striim

AUGUST 31, 2023

Showing how Kappa unifies batch and streaming pipelines The development of Kappa architecture has revolutionized data processing by allowing users to quickly and cost-effectively reduce data integration costs. Stream processors, storage layers, message brokers, and databases make up the basic components of this architecture.

Data Integration 69

Data Integration Architecture Amazon Web Services Machine Learning 69

Data Engineering Podcast

JULY 3, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. does exactly that. does exactly that.

Data Integration 100

Data Integration MongoDB MySQL Scala 100

Hevo

DECEMBER 26, 2023

Due to Spring Framework’s rich feature set, developers often face complexity while configuring Spring applications. To safeguard developers from this tedious and error-prone process, the Spring team launched Spring Boot as a useful extension of the Spring framework.

MySQL 98

MySQL Database Process Java 98

Snowflake

JANUARY 23, 2023

Marketing data integration is the process of combining marketing data from different sources to create a unified and consistent view. If you’re running marketing campaigns on multiple platforms—Facebook, Instagram, TikTok, email—you need marketing data integration. What Problems does Data Integration Solve?

Data Integration 71

Data Integration Data Data Warehouse Cloud 71

Data Engineering Podcast

APRIL 10, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Integration 100

Data Integration Data Engineer Data Engineering SQL 100

Ascend.io

NOVEMBER 14, 2024

Ensure the provider supports the infrastructure necessary for your data needs, such as managed databases, storage, and data pipeline services. Utilize Cloud-Native Tools: Leverage cloud-native data pipeline tools like Ascend to build and orchestrate scalable workflows.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Striim

JUNE 18, 2024

In today’s fast-paced world, staying ahead of the competition requires making decisions informed by the freshest data available — and quickly. That’s where real-time data integration comes into play. What is Real-Time Data Integration + Why is it Important? Why is Real-Time Data Integration Important?

Data Integration 52

Data Integration Data Lake Retail Healthcare 52

Striim

MARCH 21, 2025

Business transactions captured in relational databases are critical to understanding the state of business operations. Since the value of data quickly drops over time, organizations need a way to analyze data as it is generated. Traditionally, businesses used batch-based approaches to move data once or several times a day.

IT 52

IT Data Lake Relational Database Data Warehouse 52

Data Engineering Podcast

MARCH 5, 2023

TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. But don't worry, there is a better way.

Building 147

Building Machine Learning Database Design Metadata 147

Striim

JUNE 6, 2024

For your organization’s data integration and streaming initiatives to succeed, meeting latency requirements is crucial. Low latency, defined by the rapid transmission of data with minimal delay, is essential for maximizing the effectiveness of your data strategy. Here’s what you need to know.

Data Integration 52

Data Integration Data Ingestion Healthcare Data Pipeline 52

Snowflake

APRIL 13, 2025

The need for agentic AI in data management Traditional data management methods are increasingly insufficient given the exponential data growth. Many enterprises face overwhelming data sources, from structured databases to unstructured social media feeds. Manual processes can be time-consuming and error-prone.

Data Management 89

Data Management Management Consulting Unstructured Data 89

Striim

MARCH 21, 2025

Today were going to talk about five streaming cloud integration use cases. Streaming cloud integration moves data continuously in real time between heterogeneous databases, with in-flight data processing. You have your legacy database and you want to move it to the cloud.

Cloud 52

Cloud Database Architecture BI 52

Striim

MARCH 21, 2025

Today were going to talk about five streaming cloud integration use cases. Streaming cloud integration moves data continuously in real time between heterogeneous databases, with in-flight data processing. You have your legacy database and you want to move it to the cloud.

Cloud 52

Cloud Database Architecture BI 52

Data Engineering Weekly

JANUARY 15, 2025

They are responsible for designing, implementing, and maintaining robust, scalable data pipelines that transform raw unstructured data—text, images, videos, and more—into high-quality, AI-ready datasets. Validate synthetic data to ensure it is representative, diverse, and suitable for the intended AI applications.

Data Engineer 62

Data Engineer Data Engineering Unstructured Data Engineering 62

Knowledge Hut

MARCH 27, 2024

Think of a database as a smart, organized library that stores and manages information efficiently. On the other hand, data structures are like the tools that help organize and arrange data within a computer program. What is a Database? SQL, or structured query language, is widely used for writing and querying data.

Database 52

Database Relational Database Algorithm Data Storage 52

Ascend.io

OCTOBER 28, 2024

Filling in missing values could involve leveraging other company data sources or even third-party datasets. The cleaned data would then be stored in a centralized database, ready for further analysis. This ensures that the sales data is accurate, reliable, and ready for meaningful analysis.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Hevo

AUGUST 23, 2024

With growing data and business needs, having an efficient data integration tool to migrate and manage your data has become crucial. Almost every organization keeps its data in different locations, from the internal database to the SaaS platform.

Data Integration 52

Data Integration Finance Database Data 52

Striim

NOVEMBER 11, 2024

Challenges The retailer’s legacy data infrastructure presented significant hurdles, preventing the company from achieving its modernization goals. The goal was to consolidate data replication efforts and improve supply chain efficiency by utilizing modern cloud infrastructure.

Retail 52

Retail Database-centric Google Cloud PostgreSQL 52

Hevo

MAY 9, 2023

As a business grows, the demand to efficiently handle and process the exponentially growing data also rises. A popular open-source relational database used by several organizations across the world is PostgreSQL.

PostgreSQL 97

PostgreSQL Relational Database Database Data Integration 97

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. With the growth in tools that are focused on batch-oriented data integration and transformation, what are the reasons that an organization should still invest in streaming?

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Weekly

AUGUST 6, 2024

Instead of relying on separate analytical databases, many organizations embed analytics directly within their data warehouses. This approach simplifies data architecture and enhances performance by reducing data movement and latency. Challenges: Modify the database schema to include timestamp columns.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Monte Carlo

JUNE 24, 2024

Choosing the Right Data Type Domain Integrity Constraints How to Implement Domain Integrity Handling Exceptions and Errors in Domain Integrity Automate Monitoring of Domain Integrity with Monte Carlo What is Domain Integrity? It is a key part of data integrity alongside entity and referential integrity.

Database 52

Database MySQL SQL Datasets 52

Hevo

MAY 3, 2024

Data integration is an essential task in most organizations. The reason is that many organizations are generating huge volumes of data. This data is not always stored in a single location, but in different locations including in on-premise databases and in the cloud.

Data Integration 52

Data Integration Data Database Cloud 52

Knowledge Hut

APRIL 23, 2024

We live in a data-driven culture where familiarity with databases is crucial. Database management is crucial for businesses of all sizes to guarantee that their data is complete, safe, and easily available when needed. There will likely be a greater need for database specialists' skills in 2024.

Database 52

Database Management Relational Database SQL 52

Wayne Yaddow

MARCH 28, 2025

Unleashing GenAIEnsuring Data Quality at Scale (Part1) Transitioning from isolated repository systems to consolidated AI LLM pipelines Photo by Joshua Sortino on Unsplash Introduction This blog is based on insights from articles in Database Trends and Applications, Feb/Mar 2025 ( DBTA Journal ).

Government 52

Government Data Governance Data Data Integration 52