Striim

MARCH 21, 2025

Batch processing: data is typically extracted from databases at the end of the day, saved to disk for transformation, and then loaded in batch to a data warehouse. Batch data integration is useful for data that isn’t extremely time-sensitive. Electric bills are a relevant example.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Silectis

JANUARY 22, 2021

For those using a robust analytics database, such as the Snowflake® Data Cloud , adding the power of a data engineering platform can help maximize the value you’re getting out of that database. Data Warehouses Have Boundaries Data warehouses do what they’re meant to, they provide a high-performance environment for data analytics.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Ascend.io

OCTOBER 28, 2024

Tools like Python’s requests library or ETL/ELT tools can facilitate data enrichment by automating the retrieval and merging of external data. Read More: Discover how to build a data pipeline in 6 steps Data Integration Data integration involves combining data from different sources into a single, unified view.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Data Engineering Podcast

DECEMBER 18, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 100

Data Lake Data Warehouse Data Pipeline MongoDB 100

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

Striim

AUGUST 14, 2024

When implemented effectively, smart data pipelines seamlessly integrate data from diverse sources, enabling swift analysis and actionable insights. They empower data analysts and business users alike by providing critical information while protecting sensitive production systems. What is a Smart Data Pipeline?

Data Pipeline 52

Data Pipeline Data Architecture Transportation 52

Data Engineering Podcast

NOVEMBER 27, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100

Data Engineering Podcast

MARCH 16, 2020

Are you spending too much time maintaining your data pipeline? Snowplow empowers your business with a real-time event data pipeline running in your own cloud account without the hassle of maintenance. Set up a demo and mention you’re a listener for a special offer!

Building 100

Building Data Warehouse NoSQL Data Lake 100

Monte Carlo

AUGUST 6, 2024

Data warehouses are the centralized repositories that store and manage data from various sources. They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. However, beneath their surface lies a host of invisible risks embedded within the data warehouse layers.

Data Warehouse 52

Data Warehouse Raw Data Machine Learning BI 52

Data Engineering Podcast

DECEMBER 4, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

JUNE 19, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 6, 2022

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

Data Engineering Podcast

NOVEMBER 20, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Data Engineering Podcast

JULY 17, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

DECEMBER 11, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 30, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Engineering 100

Engineering MongoDB MySQL Scala 100

Striim

MAY 9, 2025

Introduction This recipe shows how you can build a data pipeline to read data from ServiceNow and write to BigQuery. Benefits Striims unified data streaming platform empowers organizations to infuse real-time data into AI, analytics, customer experiences and operations.

Data Pipeline 40

Data Pipeline Data Datasets Data Warehouse 40

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

APRIL 24, 2022

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Struggling with broken pipelines?

Data Warehouse 100

Data Warehouse Data Lake Data Pipeline BI 100

Cloudera

JANUARY 20, 2021

Most of what is written though has to do with the enabling technology platforms (cloud or edge or point solutions like data warehouses) or use cases that are driving these benefits (predictive analytics applied to preventive maintenance, financial institution’s fraud detection, or predictive health monitoring as examples) not the underlying data.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Data Engineering Podcast

DECEMBER 21, 2020

Fortunately, there’s hope: in the same way that New Relic, DataDog, and other Application Performance Management solutions ensure reliable software and keep application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. The first 25 will receive a free, limited edition Monte Carlo hat!

Data Governance 130

Data Governance Government Data Lake Data 130

Data Engineering Podcast

JUNE 26, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to reduce repetitive work for common workflows by adopting a visual pipeline builder to support your data warehouse transformations.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Data Validation 52

Data Engineering Podcast

FEBRUARY 24, 2020

Are you spending too much time maintaining your data pipeline? Snowplow empowers your business with a real-time event data pipeline running in your own cloud account without the hassle of maintenance. Set up a demo and mention you’re a listener for a special offer!

IT 100

IT Data Lake Data Pipeline Media 100

Data Engineering Podcast

JULY 3, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

MAY 25, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Database 100

Database Building Data Warehouse Machine Learning 100

Data Engineering Podcast

OCTOBER 2, 2021

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Building 100

Building BI Data Architecture Architecture 100

Data Engineering Podcast

APRIL 17, 2022

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

SQL 100

SQL Data Warehouse Data Engineering Data Engineer 100

Data Engineering Podcast

MARCH 2, 2020

Are you spending too much time maintaining your data pipeline? Snowplow empowers your business with a real-time event data pipeline running in your own cloud account without the hassle of maintenance. Set up a demo and mention you’re a listener for a special offer!

Kafka 100

Kafka Process PostgreSQL MySQL 100

Monte Carlo

MAY 26, 2023

Pro-tip : Don’t confuse data freshness with data latency. Data latency is the time between when the event occurs and when the data is available in the core data system (like a data warehouse) whereas data freshness is how recently the data within the final asset (table, BI report) has been updated.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Monte Carlo

JULY 9, 2024

As more organizations race to adopt GenAI and build AI-powered data products, DuckDB provides emerging applications as the storage layer of RAG knowledge bases to streamline and expedite data management. MotherDuck turbocharged DuckDB’s efficiency with multiplayer cloud analytics, making it a lightweight but powerful data warehouse.

Data Warehouse 69

Data Warehouse Data Pipeline SQL Database 69

Data Engineering Podcast

JULY 10, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Machine Learning 100

Data Engineering Podcast

DECEMBER 16, 2019

You work hard to make sure that your data is clean, reliable, and reproducible throughout the ingestion pipeline, but what happens when it gets to the data warehouse? Dataform picks up where your ETL jobs leave off, turning raw data into reliable analytics.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

MARCH 27, 2022

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Relational Database 100

Relational Database Database SQL Data Warehouse 100

Data Engineering Podcast

JANUARY 4, 2021

Fortunately, there’s hope: in the same way that New Relic, DataDog, and other Application Performance Management solutions ensure reliable software and keep application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. The first 25 will receive a free, limited edition Monte Carlo hat!

Python 100

Python Machine Learning Computer Science Data Lake 100

Data Engineering Podcast

JANUARY 11, 2021

Fortunately, there’s hope: in the same way that New Relic, DataDog, and other Application Performance Management solutions ensure reliable software and keep application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. The first 25 will receive a free, limited edition Monte Carlo hat!

PostgreSQL 100

PostgreSQL Python Computer Science Data Lake 100