Christophe Blefari

JANUARY 11, 2025

The Data News are here to stay, the format might vary during the year, but here we are for another year. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. HNY 2025 ( credits ) Happy new year ✨ I wish you the best for 2025. Not really digest.

Data 130

Data Data Warehouse Coding Programming Language 130

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 140

Data Lake Data Warehouse Data Storage Data 140

Seattle Data Guy

FEBRUARY 1, 2023

Recently several consulting calls started with people asking, “Do we need a data warehouse?” ” This isn’t a question about whether you need data warehouse consultants, but instead whether you should event start a data warehouse project. Not every company needs a data warehouse.

Data Warehouse 130

Data Warehouse Consulting Data Project 130

Monte Carlo

FEBRUARY 11, 2025

Think of your data warehouse like a well-organized library. Thats where data warehouse schemas come in. A data warehouse schema is a blueprint for how your data is structured and linkedusually with fact tables (for measurable data) and dimension tables (for descriptive attributes). Total chaos.

Data Warehouse 52

Data Warehouse Electronics Retail Data 52

Seattle Data Guy

JANUARY 12, 2023

A data warehouse consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. But in order to understand why you need a data warehouse consultant we should take a step back.

Consulting 130

Consulting Data Warehouse Data Designing 130

Start Data Engineering

APRIL 12, 2022

Introduction Data lakes and data warehouses Data lake Data warehouse Criteria to choose lake and warehouse tools Conclusion Further reading References Introduction With the data ecosystem growing fast, new terms are coming up every week.

Data Lake 130

Data Lake Data Warehouse Data 130

Seattle Data Guy

DECEMBER 19, 2022

By Reseun McClendon Today, your enterprise must effectively collect, store, and integrate data from disparate sources to both provide operational and analytical benefits. Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Towards Data Science

JANUARY 28, 2024

How I organize everything in my new data warehouse Continue reading on Towards Data Science »

Data Warehouse 98

Data Warehouse Designing Data Science Data 98

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Simon Späti

MARCH 11, 2025

Many data engineers and analysts start their journey with Postgres. But data volumes grow, analytical demands become more complex, and Postgres stops being enough. But data volumes grow, analytical demands become more complex, and Postgres stops being enough.

Database 130

Database Data Warehouse Data Engineering Data Engineer 130

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6.

Data Engineering 246

Data Engineering Data Engineer Project Data Lake 246

databricks

FEBRUARY 4, 2025

Databricks welcomes BladeBridge, a proven provider of AI-powered migration solutions for enterprise data warehouses. Together, Databricks and BladeBridge will help enterprises accelerate the.

Data Warehouse 115

Data Warehouse Data 115

Confessions of a Data Guy

OCTOBER 7, 2024

Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home? Can we say that after all these decades the Data Warehouse Toolkit […] The post The Death of the Data Warehouse, replaced by the Lake House.

Data Warehouse 113

Data Warehouse IT Data Big Data 113

Data Engineering Podcast

MARCH 10, 2023

NetSpring is a warehouse-native product analytics service that allows you to gain powerful insights into your customers and their needs by combining your event streams with the rest of your business data. Visit: dataengineeringpodcast.com/data-council today! Don't miss out on their only event this year!

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Towards Data Science

JULY 30, 2024

Rethinking data warehousing: Why redefinition is necessary even beyond Modern Data Warehouse (MDW) and Lakehouse Models Continue reading on Towards Data Science »

Data Warehouse 75

Data Warehouse Data Science Data Data Architecture 75

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 107

Data Lake Data Warehouse Cloud Unstructured Data 107

Christophe Blefari

MARCH 15, 2024

Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. Pandera, a data validation library for dataframes, now supports Polars.

Metadata 272

Metadata Data Datasets Data Warehouse 272

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

FEBRUARY 24, 2024

Best practice and advanced techniques for beginners Continue reading on Towards Data Science »

Data Warehouse 98

Data Warehouse Building Data Science Data 98

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 78

Data Lake Cloud Storage Metadata Data Warehouse 78

Confessions of a Data Guy

MAY 30, 2024

Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending.

Building 184

Building Data Warehouse Data Data Engineering 184

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessible 162

Hevo

SEPTEMBER 4, 2024

A data warehouse is a centralized system that stores, integrates, and analyzes large volumes of structured data from various sources. It is predicted that more than 200 zettabytes of data will be stored in the global cloud by 2025.

Data Warehouse 72

Data Warehouse Structured Data Data Cloud 72

Monte Carlo

JANUARY 14, 2025

When data engineers tell scary stories around a campfire, its usually a cautionary tale about the cost of poor data quality. Data downtime can occur suddenly at any timeand often not when or where youre looking for it. But just how much can data downtime actually cost your business?

Data 98

Data Data Engineering Data Engineer Media 98

Analytics Vidhya

FEBRUARY 1, 2023

Introduction Nowadays, organizations are looking for multiple solutions to deal with big data and related challenges.

Data Cleanse 223

Data Cleanse Data Warehouse Big Data Cloud 223

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Snowflake

MARCH 12, 2025

In todays data-driven world, organizations depend on high-quality data to drive accurate analytics and machine learning models. But poor data quality gaps, inconsistencies and errors can undermine even the most sophisticated data and AI initiatives.

Unstructured Data 77

Unstructured Data High Quality Data Banking Machine Learning 77

Engineering at Meta

FEBRUARY 4, 2025

Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand.

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Towards Data Science

DECEMBER 10, 2024

Three Zero-Cost Solutions That Take Hours, NotMonths A data quality certified pipeline. Source: unsplash.com In my career, data quality initiatives have usually meant big changes. Whats more, fixing the data quality issues this way often leads to new problems. Create a custom dashboard for your specific data qualityproblem.

PostgreSQL 75

PostgreSQL Data Python SQL 75

Analytics Vidhya

FEBRUARY 8, 2023

Introduction Amazon Redshift is a fully managed, petabyte-scale data warehousing Amazon Web Services (AWS). It allows users to easily set up, operate, and scale a data warehouse in the cloud.

Amazon Web Services 227

Amazon Web Services AWS Data Warehouse Cloud 227

Snowflake

DECEMBER 4, 2024

Together with a dozen experts and leaders at Snowflake, I have done exactly that, and today we debut the result: the “ Snowflake Data + AI Predictions 2024 ” report. When you’re running a large language model, you need observability into how the model may change as it ingests new data. The next evolution in data is making it AI ready.

Unstructured Data 86

Unstructured Data Data Lake Deep Learning Structured Data 86

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 258

Data Engineering Data Engineer Engineering Data 258

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Weekly

NOVEMBER 24, 2024

Editor’s Note: Launching Data & Gen-AI courses in 2025 I can’t believe DEW will reach almost its 200th edition soon. What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. We are planning many exciting product lines to trial and launch in 2025.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57