Data Engineering Weekly

NOVEMBER 24, 2024

Editor’s Note: Launching Data & Gen-AI courses in 2025 I can’t believe DEW will reach almost its 200th edition soon. What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. We are planning many exciting product lines to trial and launch in 2025.

Data Engineer 57

Data Engineer Data Engineering Engineering Insurance 57

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineer 251

Data Engineer Data Engineering Engineering Data 251

Data Engineering Podcast

MARCH 10, 2023

NetSpring is a warehouse-native product analytics service that allows you to gain powerful insights into your customers and their needs by combining your event streams with the rest of your business data. Visit: dataengineeringpodcast.com/data-council today! Don't miss out on their only event this year!

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Seattle Data Guy

FEBRUARY 1, 2023

Recently several consulting calls started with people asking, “Do we need a data warehouse?” ” This isn’t a question about whether you need data warehouse consultants, but instead whether you should event start a data warehouse project. Not every company needs a data warehouse.

Data Warehouse 130

Data Warehouse Consulting Data Project 130

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineer 225

Data Engineer Data Engineering Python Engineering 225

Start Data Engineering

OCTOBER 3, 2021

What is a data warehouse 4. OLTP vs OLAP based data warehouses 5. Introduction If you are a student, analyst, engineer, or anyone in the data space, it’s important to understand what a data warehouse is. If you are wondering What is a data warehouse? Conclusion 6.

Data Warehouse 130

Data Warehouse Data Engineering IT 130

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6.

Data Engineer 246

Data Engineer Data Engineering Project Data Lake 246

Towards Data Science

JULY 30, 2024

Rethinking data warehousing: Why redefinition is necessary even beyond Modern Data Warehouse (MDW) and Lakehouse Models Continue reading on Towards Data Science »

Data Warehouse 57

Data Warehouse Data Science Data Data Architecture 57

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Materialize]([link] Looking for the simplest way to get the freshest data possible to your teams?

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Seattle Data Guy

JANUARY 12, 2023

A data warehouse consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. But in order to understand why you need a data warehouse consultant we should take a step back.

Consulting 130

Consulting Data Warehouse Data Designing 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 131

Data Lake Data Warehouse Data Storage Data 131

Towards Data Science

JANUARY 28, 2024

How I organize everything in my new data warehouse Continue reading on Towards Data Science »

Data Warehouse 65

Data Warehouse Designing Data Science Data 65

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly. Apache Kafka came in 2011 and gave the industry a much better way to move real-time data.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Weekly

OCTOBER 6, 2024

Minimize the data used in pipelines, aka do incremental data pipeline design. Filter data effectively to make sure the query uses partition pruning. Don’t query two years of data. Data quality rules, that is. Regularly review if pipelines are still required. Optimize pipeline schedules.

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62

Seattle Data Guy

DECEMBER 19, 2022

By Reseun McClendon Today, your enterprise must effectively collect, store, and integrate data from disparate sources to both provide operational and analytical benefits. Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Start Data Engineering

APRIL 12, 2022

Introduction Data lakes and data warehouses Data lake Data warehouse Criteria to choose lake and warehouse tools Conclusion Further reading References Introduction With the data ecosystem growing fast, new terms are coming up every week.

Data Lake 130

Data Lake Data Warehouse Data 130

Start Data Engineering

AUGUST 17, 2021

Batch Data Pipelines 1.1 Process => Data Warehouse 1.2 Process => Cloud Storage => Data Warehouse 2. Near Real-Time Data pipelines 2.1 Data Stream => Consumer => Data Warehouse 2.2 Near Real-Time Data pipelines 2.1 Data Stream => Consumer => Data Warehouse 2.2

Data Warehouse 130

Data Warehouse Cloud Storage Data Pipeline Data 130

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Warehouse 130

Data Engineering Podcast

MAY 13, 2021

Summary There is a lot of attention on the database market and cloud data warehouses. While they provide a measure of convenience, they also require you to sacrifice a certain amount of control over your data. Firebolt is the fastest cloud data warehouse. Visit dataengineeringpodcast.com/firebolt to get started.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Warehouse 81

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

Data Engineering Weekly

SEPTEMBER 22, 2024

Editor’s Note: Coming Next on Comparison Matrix Series - Data LakeHouse Our mission is to empower data professionals and organizations to make informed, data-driven decisions by providing a comprehensive buyer's guide and comparison matrix for selecting the best data tools.

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Data Engineering Weekly

MAY 26, 2024

link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. seconds, enhancing real-time sports data analytics efficiency! The author highlights some of the key missing features from S3.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Data Engineering Weekly

SEPTEMBER 2, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 67

Data Engineer Data Engineering Engineering Entertainment 67

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 94

Data Lake Data Warehouse Cloud Unstructured Data 94

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Hudi seems to be a de facto choice for CDC data lake features. Notion migrated the insert heavy workload from Snowflake to Hudi.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Hepta Analytics

FEBRUARY 24, 2022

Week 3 was about data warehousing, working on the data that was ingested in the week 2. We will take the already ingested data and create an external table from it and optimize the performance of queries through partitioning and clustering. Data is stored in traditional databases. Mostly for high volume customers.

Data Warehouse 52

Data Warehouse Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

JANUARY 1, 2022

Summary Communication and shared context are the hardest part of any data system. In recent years the focus has been on data catalogs as the means for documenting data assets, but those introduce a secondary system of record in order to find the necessary information.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Engineer 100

Towards Data Science

MAY 8, 2023

Data Warehouse Dimensional Modeling Design 101 Continue reading on Towards Data Science »

Data Warehouse 66

Data Warehouse Data Engineering Data Engineer Building 66

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses.

Data Engineer 58

Data Engineer Data Engineering Engineering Data Warehouse 58

Data Engineering Podcast

FEBRUARY 8, 2021

Summary With all of the tools and services available for building a data platform it can be difficult to separate the signal from the noise. In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify.

Data Warehouse 100

Data Warehouse Building BI SQL 100

Confessions of a Data Guy

MAY 30, 2024

Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending.

Building 184

Building Data Warehouse Data Data Engineer 184

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. I bet you are, this article is for you!

Data Engineer 82

Data Engineer Data Engineering Engineering Data Lake 82