Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Data Engineering Podcast

OCTOBER 28, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Start trusting your data with Monte Carlo today!

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Striim

APRIL 18, 2025

But theyre only as good as the data they rely on. If the underlying data is incomplete, inconsistent, or delayed, even the most advanced AI models and business intelligence systems will produce unreliable insights. Heres why: AI Models Require Clean Data: Machine learning models are only as good as their training data.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Monte Carlo

JANUARY 20, 2024

The future of business intelligence (BI) is inextricably linked to the future of data. As the amount of data companies create and consume grows exponentially, the speed and ease with which you can access and rely upon that data is going to be more important than ever before.

Business Intelligence 52

Business Intelligence BI Raw Data Datasets 52

Towards Data Science

OCTOBER 25, 2023

As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. You probably work closely with the colleague who builds the data pipeline that you were consuming. There was only one data team, two at most. null, null).

Data Pipeline 98

Data Pipeline SQL Data Architecture Data 98

Ascend.io

OCTOBER 28, 2024

Data Aggregation Data aggregation is a powerful technique that involves compiling data from various sources to provide a comprehensive view. This process is crucial for generating summary statistics, such as averages, sums, and counts, which are essential for business intelligence and analytics.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 130

Data Engineering Data Engineer Engineering Project 130

ProjectPro

JANUARY 31, 2023

Want to know who is a business intelligence engineer, what does a business intelligence engineer do, and how these BI engineers turn mountains of data into actionable insights? According to Fortune Business Insights, the global market for business intelligence is likely to grow at a CAGR of 8.7%

Business Intelligence 52

Business Intelligence Engineering BI Data Warehouse 52

Data Engineering Podcast

DECEMBER 18, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

DECEMBER 29, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.

Management 130

Management Metadata Business Intelligence Data Lake 130

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

Team Data Science

NOVEMBER 4, 2020

After the boot camp, as I was searching for Data Science opportunities in the industry, I noticed the requirements for a Data Scientist has changed from what it was 2-3 years ago. My goal for this course is to create a complete end-to-end data pipeline for an application that is time series related.

Data Science 130

Data Science Machine Learning Programming Language Business Intelligence 130

Monte Carlo

APRIL 24, 2023

Build vs buy orchestration tooling Unlike the other components we’ve discussed in Part 3, data pipelines don’t require orchestration to be considered functional—at least not at a foundational level. And data orchestration tools are generally easy to stand-up for initial use-cases. Missed Nishith’s 5 considerations?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Data Engineering Podcast

JANUARY 8, 2023

Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box.

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Data Engineering Podcast

JANUARY 15, 2023

Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box.

Coding 130

Coding Building PostgreSQL Data Lake 130

DataKitchen

NOVEMBER 5, 2024

Finally, the Gold laye r represents the pinnacle of the Medallion architecture, housing fully refined, aggregated, and analysis-ready data. Data is typically organized into project-specific schemas optimized for business intelligence (BI) applications, advanced analytics, and machine learning.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Data Engineering Weekly

FEBRUARY 2, 2025

Shifting left involves moving data processing upstream, closer to the source, enabling broader access to high-quality data through well-defined data products and contracts, thus reducing duplication, enhancing data integrity, and bridging the gap between operational and analytical data domains.

Data Engineering 58

Data Engineering Data Engineer Engineering Data Lake 58

Data Engineering Podcast

NOVEMBER 20, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. All it takes is 3 steps for your pipeline to be up and running.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

APRIL 26, 2021

Summary The reason for collecting, cleaning, and organizing data is to make it usable by the organization. One of the most common and widely used methods of access is through a business intelligence dashboard. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Data Engineering Podcast

NOVEMBER 13, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.

Building 130

Building Metadata MongoDB MySQL 130

Monte Carlo

JULY 19, 2023

Like bean dip and ogres , layers are the building blocks of the modern data stack. Its powerful selection of tooling components combine to create a single synchronized and extensible data platform with each layer serving a unique function of the data pipeline. Let’s dive into it. The content, not the bean dip.

Building 96

Building Business Intelligence Cloud Storage BI 96

Ascend.io

APRIL 28, 2023

Data pipelines are having a moment — at least, that is, within the data world. That’s because as more and more businesses are adopting a data-driven mindset, the movement of data into and within organizations has never been a bigger priority.

Data Pipeline 52

Data Pipeline Data Data Warehouse Data Engineer 52

Edureka

APRIL 22, 2025

Snowflake: Architecture Microsoft Fabric Architecture Azure is the foundation of Microsoft Fabric, a Software-as-a-Service (SaaS) data platform. Data integration, data engineering, data warehousing, real-time analytics, data science, and business intelligence are among the analytics tasks it unifies into a single, cohesive interface.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Towards Data Science

JULY 21, 2023

Like bean dip and ogres , layers are the building blocks of the modern data stack. Its powerful selection of tooling components combine to create a single synchronized and extensible data platform with each layer serving a unique function of the data pipeline. Let’s dive into it. The content, not the bean dip.

Building 98

Building Business Intelligence BI Cloud Storage 98

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Data Engineering Podcast

NOVEMBER 27, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.

Data Process 100

Data Process Process Metadata Business Intelligence 100

Ascend.io

JULY 28, 2023

Data pipelines are integral to business operations, regardless of whether they are meticulously built in-house or assembled using various tools. As companies become more data-driven, the scope and complexity of data pipelines inevitably expand. Ready to fortify your data management practice?

Data Pipeline 52

Data Pipeline Architecture Lambda Architecture Data Architecture 52

Data Engineering Podcast

OCTOBER 22, 2021

Summary The market for business intelligence has been going through an evolutionary shift in recent years. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. Get started for free at dataengineeringpodcast.com/hightouch.

Engineering 100

Engineering Business Intelligence Data Warehouse BI 100

Data Engineering Podcast

NOVEMBER 6, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.

Building 130

Building IT Metadata MongoDB 130

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Edureka

APRIL 14, 2025

This blog breaks down how these tools complement and differ from one another to help you identify the best fit for your business. Understanding the Tools One platform is designed primarily for business intelligence, offering intuitive ways to connect to various data sources, build interactive dashboards, and share insights.

BI 40

BI Business Intelligence Raw Data Retail 40

Data Engineering Podcast

JULY 17, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Tired of deploying bad data? Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

JANUARY 15, 2022

Summary Applications of data have grown well beyond the venerable business intelligence dashboards that organizations have relied on for decades. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud.

Engineering 100

Engineering Electronics Data Pipeline ETL Tools 100

Data Engineering Podcast

DECEMBER 25, 2022

Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value.

Building 100

Building Metadata Business Intelligence Data Lake 100

Towards Data Science

NOVEMBER 4, 2023

I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)? Are your data pipelines efficient? What is it? The downside of this approach is it’s pricing model though.

Data Engineering 98

Data Engineering Data Engineer Engineering BI 98

Maxime Beauchemin

FEBRUARY 28, 2017

A few weeks ago it was The Rise of the Data Engineer by Maxime Beauchemin, a data engineer at Airbnb and creator of their data pipeline framework, Apache Airflow. You might be wondering, “What is data engineering and why does it matter?”

Data Engineering 130

Data Engineering Data Engineer Engineering Data Pipeline 130