Start Data Engineering

OCTOBER 3, 2021

What is a data warehouse 4. OLTP vs OLAP based data warehouses 5. Introduction If you are a student, analyst, engineer, or anyone in the data space, it’s important to understand what a data warehouse is. If you are wondering What is a data warehouse? Conclusion 6.

Data Warehouse 130

Data Warehouse Data Engineering IT 130

Simon Späti

MARCH 11, 2025

Many data engineers and analysts start their journey with Postgres. But data volumes grow, analytical demands become more complex, and Postgres stops being enough. But data volumes grow, analytical demands become more complex, and Postgres stops being enough.

Database 130

Database Data Warehouse Data Engineering Data Engineer 130

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Support Data Engineering Podcast RudderStack also supports real-time use cases.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Analytics Vidhya

APRIL 3, 2023

Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.

Data Engineering 258

Data Engineering Data Engineer Engineering Data 258

Precisely

JANUARY 9, 2025

The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s.

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

Engineering at Meta

JANUARY 22, 2025

In order to build high-quality data lineage, we developed different techniques to collect data flow signals across different technology stacks: static code analysis for different languages, runtime instrumentation, and input and output data matching, etc. PyTorch ), workflow engines (e.g., Hack, C++, Python, etc.)

Data Warehouse 120

Data Warehouse SQL Programming Language Data 120

Data Engineering Weekly

NOVEMBER 24, 2024

Editor’s Note: Launching Data & Gen-AI courses in 2025 I can’t believe DEW will reach almost its 200th edition soon. What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. We are planning many exciting product lines to trial and launch in 2025.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Christophe Blefari

JANUARY 11, 2025

A few things to notice: Interest in AI grew by 190%, Prompt Engineering by 456%. Hybrid Kimball & OBT data modeling approach — This is maybe the most common setup I've seen the last 3 years. Analytics engineering at Netflix — (and part 2 ). From the traffic they get they draw market trends.

Data 130

Data Data Warehouse Coding Programming Language 130

Seattle Data Guy

FEBRUARY 1, 2023

Recently several consulting calls started with people asking, “Do we need a data warehouse?” ” This isn’t a question about whether you need data warehouse consultants, but instead whether you should event start a data warehouse project. Not every company needs a data warehouse.

Data Warehouse 130

Data Warehouse Consulting Data Project 130

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Start Data Engineering

AUGUST 17, 2021

Batch Data Pipelines 1.1 Process => Data Warehouse 1.2 Process => Cloud Storage => Data Warehouse 2. Near Real-Time Data pipelines 2.1 Data Stream => Consumer => Data Warehouse 2.2

Data Warehouse 130

Data Warehouse Cloud Storage Data Pipeline Data 130

Data Engineering Podcast

MARCH 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Seattle Data Guy

JANUARY 12, 2023

A data warehouse consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. But in order to understand why you need a data warehouse consultant we should take a step back.

Consulting 130

Consulting Data Warehouse Data Designing 130

Seattle Data Guy

DECEMBER 19, 2022

Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 135

Data Lake Data Warehouse Data Storage Data 135

Start Data Engineering

APRIL 12, 2022

Introduction Data lakes and data warehouses Data lake Data warehouse Criteria to choose lake and warehouse tools Conclusion Further reading References Introduction With the data ecosystem growing fast, new terms are coming up every week.

Data Lake 130

Data Lake Data Warehouse Data 130

Towards Data Science

JANUARY 28, 2024

How I organize everything in my new data warehouse Continue reading on Towards Data Science »

Data Warehouse 98

Data Warehouse Designing Data Science Data 98

Towards Data Science

MAY 8, 2023

Data Warehouse Dimensional Modeling Design 101 Continue reading on Towards Data Science »

Data Warehouse 96

Data Warehouse Data Engineering Data Engineer Building 96

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 113

Data Lake Data Warehouse Cloud Unstructured Data 113

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit [dataengineeringpodcast.com/data-council]([link] and use code *depod20* to register today!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Warehouse 130

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 93

Data Lake Cloud Storage Metadata Data Warehouse 93

Towards Data Science

JULY 30, 2024

Rethinking data warehousing: Why redefinition is necessary even beyond Modern Data Warehouse (MDW) and Lakehouse Models Continue reading on Towards Data Science »

Data Warehouse 75

Data Warehouse Data Science Data Data Architecture 75

Data Engineering Weekly

FEBRUARY 2, 2025

[link] Get Your Guide: From Snowflake to Databricks: Our cost-effective journey to a unified data warehouse. GetYourGuide discusses migrating its Business Intelligence (BI) data source from Snowflake to Databricks, achieving a 20% cost reduction.

Data Engineering 57

Data Engineering Data Engineer Engineering Data Lake 57

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

APRIL 23, 2023

In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Seattle Data Guy

APRIL 21, 2025

If you work in data, then youve likely used BigQuery and youve likely used it without really thinking about how it operates under the hood. On the surface BigQuery is Google Clouds fully-managed, serverless data warehouse. … Read more The post What Is BigQuery And How Do You Load Data Into It?

IT 130

IT Google Cloud Data Warehouse Data 130

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Towards Data Science

JUNE 6, 2023

The fact it’s insanely fast and does (mostly) all processing in memory make it a good choice for building my personal data warehouse. Fitbit activity data The first collection of files I looked at was activity data. Extra points for its seamless integration with Python and R.

Data Warehouse 98

Data Warehouse Data SQL Data Analysis 98

Christophe Blefari

MARCH 15, 2024

Cognition AI introduced Devin — Devin is the first AI software engineer, Devin can, unassisted, do software engineering tasks like fixing Github issues (13% of success, previously best was ~5%), apply to jobs on Upwork, train and fine-tune its own models. Arrow doing a lot of the data operation heavy lifting.

Metadata 272

Metadata Data Software Engineer Software Engineering 272

Data Engineering Weekly

FEBRUARY 9, 2025

The blog post highlights the industry trend of search engines transitioning towards embedding-based systems, moving beyond traditional IDF models. link] Meta: Data logs - The latest evolution in Meta’s access tools Meta writes about its access tool's system design, which helps export individual users’ access logs.

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57