Data Engineering Podcast

FEBRUARY 22, 2021

Summary Data integration is a critical piece of every data pipeline, yet it is still far from being a solved problem. The team at Airbyte is adding a new entry to that list with the goal of making robust and easy to use data integration more accessible to teams who want or need to maintain full control of their data.

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Netflix Tech

OCTOBER 27, 2020

Usually Data scientists and engineers write Extract-Transform-Load (ETL) jobs and pipelines using big data compute technologies, like Spark or Presto , to process this data and periodically compute key information for a member or a video. The processed data is typically stored as data warehouse tables in AWS S3.

Data Warehouse 100

Data Warehouse Datasets Data Big Data 100

Data Engineering Podcast

NOVEMBER 30, 2020

Summary The first stage of every good pipeline is to perform data integration. With the increasing pace of change and the need for up to date analytics the need to integrate that data in near real time is growing. There are a number of projects and platforms on the market that target data integration.

Data Integration 100

Data Integration Coding BI Data Engineering 100

Data Engineering Podcast

AUGUST 27, 2021

Summary The reason that so much time and energy is spent on data integration is because of how our applications are designed. By making the software be the owner of the data that it generates, we have to go through the trouble of extracting the information to then be used elsewhere. What is Zero-Copy Integration?

Data Integration 100

Data Integration Architecture Data Warehouse Data Lake 100

Hevo

SEPTEMBER 4, 2024

A data warehouse is a centralized system that stores, integrates, and analyzes large volumes of structured data from various sources. It is predicted that more than 200 zettabytes of data will be stored in the global cloud by 2025.

Data Warehouse 72

Data Warehouse Structured Data Data Cloud 72

Data Engineering Podcast

OCTOBER 5, 2020

Summary Analytical workloads require a well engineered and well maintained data integration process to ensure that your information is reliable and up to date. Building a real-time pipeline for your data lakes and data warehouses is a non-trivial effort, requiring a substantial investment of time and energy.

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. With the improvements in streaming engines it is now possible to perform all of your data integration in near real time, but it can be challenging to understand the proper processing patterns to make that performant.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Precisely

MARCH 9, 2023

Data warehouse vs. data lake, each has their own unique advantages and disadvantages; it’s helpful to understand their similarities and differences. In this article, we’ll focus on a data lake vs. data warehouse. Read Many of the preferred platforms for analytics fall into one of these two categories.

Data Lake 59

Data Lake Data Warehouse Hadoop Raw Data 59

Data Engineering Podcast

JANUARY 25, 2021

In order to reduce the friction involved in supporting new data transformations David Molot and Hassan Syyid built the Hotlue platform. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Integration 100

Data Integration IT Python BI 100

Snowflake

JANUARY 23, 2023

Marketing data integration is the process of combining marketing data from different sources to create a unified and consistent view. If you’re running marketing campaigns on multiple platforms—Facebook, Instagram, TikTok, email—you need marketing data integration. What Problems does Data Integration Solve?

Data Integration 77

Data Integration Data Data Warehouse Cloud 77

Data Engineering Podcast

JULY 2, 2021

Summary Data integration in the form of extract and load is the critical first step of every data project. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Integration 100

Data Integration SQL BI Data Warehouse 100

Knowledge Hut

APRIL 23, 2024

Two popular approaches that have emerged in recent years are data warehouse and big data. While both deal with large datasets, but when it comes to data warehouse vs big data, they have different focuses and offer distinct advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Data Engineering Podcast

APRIL 10, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Integration 100

Data Integration Data Engineering Data Engineer SQL 100

Striim

AUGUST 31, 2023

Kappa Architecture combines streaming and batch while simultaneously turning data warehouses and data lakes into near real-time sources of truth. Overview of kappa architecture Kappa architecture is a powerful data processing architecture that enables near-real-time data processing.

Data Integration 69

Data Integration Architecture Amazon Web Services Machine Learning 69

Monte Carlo

SEPTEMBER 10, 2024

This truth was hammered home recently when ride-hailing giant Uber found itself on the receiving end of a staggering €290 million ($324 million) fine from the Dutch Data Protection Authority. Poor data warehouse governance practices that led to the improper handling of sensitive European driver data. The reason?

Data Warehouse 52

Data Warehouse Government Data Governance Metadata 52

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility.

Data Integration 100

Data Integration MongoDB MySQL Scala 100

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Data Validation 52

Striim

MARCH 21, 2025

Batch processing: data is typically extracted from databases at the end of the day, saved to disk for transformation, and then loaded in batch to a data warehouse. Batch data integration is useful for data that isn’t extremely time-sensitive. Real-time data processing has many use cases.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Monte Carlo

AUGUST 6, 2024

Data warehouses are the centralized repositories that store and manage data from various sources. They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. Integration Layer : Where your data transformations and business logic are applied.

Data Warehouse 52

Data Warehouse Raw Data Machine Learning BI 52

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. With the growth in tools that are focused on batch-oriented data integration and transformation, what are the reasons that an organization should still invest in streaming?

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Ascend.io

NOVEMBER 14, 2024

Databricks and Apache Spark provide robust parallel processing capabilities for big data workloads, making it easier to distribute tasks across multiple nodes and improve throughput. Integration: Seamless Data Integration Strategies Integrating diverse data sources is crucial for maintaining pipeline efficiency and reducing complexity.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Maxime Beauchemin

AUGUST 28, 2017

Consensus seeking Whether you think that old-school data warehousing concepts are fading or not, the quest to achieve conformed dimensions and conformed metrics is as relevant as it ever was. The data warehouse needs to reflect the business, and the business should have clarity on how it thinks about analytics.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Precisely

MAY 13, 2024

Key Takeaways: Data integration is vital for real-time data delivery across diverse cloud models and applications, and for leveraging technologies like generative AI. The right data integration solution helps you streamline operations, enhance data quality, reduce costs, and make better data-driven decisions.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Engineering Weekly

FEBRUARY 2, 2025

Shifting left involves moving data processing upstream, closer to the source, enabling broader access to high-quality data through well-defined data products and contracts, thus reducing duplication, enhancing data integrity, and bridging the gap between operational and analytical data domains.

Data Engineering 58

Data Engineering Data Engineer Engineering Data Lake 58

Striim

MARCH 21, 2025

Since the value of data quickly drops over time, organizations need a way to analyze data as it is generated. To avoid disruptions to operational databases, companies typically replicate data to data warehouses for analysis. What is Change Data Capture?

IT 52

IT Data Lake Data Warehouse Relational Database 52

Data Engineering Weekly

AUGUST 6, 2024

As a result, lakehouses support more dynamic and flexible data architectures, catering to a broader range of analytics and operational workloads. For instance, in a fast-paced retail environment, lakehouses can ensure that inventory data remains up-to-date and accurate in the data warehouse, optimizing supply chain efficiency.

Data Lake 126

Data Lake Data Warehouse Database Data Architecture 126

Striim

JUNE 18, 2024

In today’s fast-paced world, staying ahead of the competition requires making decisions informed by the freshest data available — and quickly. That’s where real-time data integration comes into play. What is Real-Time Data Integration + Why is it Important? Why is Real-Time Data Integration Important?

Data Integration 52

Data Integration Data Lake Retail Healthcare 52

Edureka

APRIL 22, 2025

Fabric is meant for organizations looking for a single pane of glass across their data estate with seamless integration and a low learning curve for Microsoft users. Snowflake is a cloud-native platform for data warehouses that prioritizes collaboration, scalability, and performance. Office 365, Power BI, Azure).

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Engineering at Meta

APRIL 28, 2025

To ensure comprehensive protection, it is essential to apply the necessary steps to all systems that store or process data, including distributed systems (web systems, chat, mobile and backend services) and data warehouses. Consider the data flow from online systems to the data warehouse, as shown in the diagram below.

Metadata 72

Metadata Data Utilities Data Warehouse 72

Hevo

MAY 10, 2024

Amazon S3 is a prominent data storage platform with multiple storage and security features. Integrating data stored in Amazon S3 to a data warehouse like Databricks can enable better data-driven decisions. Integrating data from Amazon S3 to Databricks […]

Data Integration 52

Data Integration Data Warehouse Data Storage Machine Learning 52

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Databand.ai

JULY 6, 2023

Data Integrity Testing: Goals, Process, and Best Practices Niv Sluzki July 6, 2023 What Is Data Integrity Testing? Data integrity testing refers to the process of validating the accuracy, consistency, and reliability of data stored in databases, data warehouses, or other data storage systems.

Data Integration 52

Data Integration Process Data Validation Data Governance 52

Snowflake

APRIL 23, 2024

If you’re a Snowflake customer using ServiceNow’s popular SaaS application to manage your digital workloads, data integration is about to get a lot easier — and less costly. The connector provides immediate access to up-to-date ServiceNow data without the need to manually integrate against API endpoints.

Data Warehouse 111

Data Warehouse Data Integration Consulting SQL 111