Start Data Engineering

AUGUST 17, 2021

Batch Data Pipelines 1.1 Process => Data Warehouse 1.2 Process => Cloud Storage => Data Warehouse 2. Near Real-Time Data pipelines 2.1 Data Stream => Consumer => Data Warehouse 2.2

Data Warehouse 130

Data Warehouse Cloud Storage Data Pipeline Data 130

Seattle Data Guy

JANUARY 12, 2023

A data warehouse consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. But in order to understand why you need a data warehouse consultant we should take a step back.

Consulting 130

Consulting Data Warehouse Data Designing 130

Start Data Engineering

APRIL 12, 2022

Introduction Data lakes and data warehouses Data lake Data warehouse Criteria to choose lake and warehouse tools Conclusion Further reading References Introduction With the data ecosystem growing fast, new terms are coming up every week.

Data Lake 130

Data Lake Data Warehouse Data 130

Seattle Data Guy

DECEMBER 19, 2022

Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Confessions of a Data Guy

OCTOBER 7, 2024

Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home? Can we say that after all these decades the Data Warehouse Toolkit […] The post The Death of the Data Warehouse, replaced by the Lake House.

Data Warehouse 113

Data Warehouse IT Data Big Data 113

Towards Data Science

JANUARY 28, 2024

How I organize everything in my new data warehouse Continue reading on Towards Data Science »

Data Warehouse 82

Data Warehouse Designing Data Science Data 82

Data Engineering Podcast

MARCH 10, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Snowflake

NOVEMBER 26, 2024

Many of our customers — from Marriott to AT&T — start their journey with the Snowflake AI Data Cloud by migrating their data warehousing workloads to the platform. Today we’re focusing on customers who migrated from a cloud data warehouse to Snowflake and some of the benefits they saw.

Data Warehouse 86

Data Warehouse Cloud PostgreSQL Hadoop 86

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 97

Data Lake Data Warehouse Cloud Unstructured Data 97

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Next steps 9.

Data Engineering 246

Data Engineering Data Engineer Project Data Lake 246

Hevo

SEPTEMBER 4, 2024

A data warehouse is a centralized system that stores, integrates, and analyzes large volumes of structured data from various sources. It is predicted that more than 200 zettabytes of data will be stored in the global cloud by 2025.

Data Warehouse 72

Data Warehouse Structured Data Data Cloud 72

Analytics Vidhya

FEBRUARY 1, 2023

Introduction Nowadays, organizations are looking for multiple solutions to deal with big data and related challenges.

Data Cleanse 216

Data Cleanse Data Warehouse Big Data Cloud 216

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Analytics Vidhya

FEBRUARY 8, 2023

Introduction Amazon Redshift is a fully managed, petabyte-scale data warehousing Amazon Web Services (AWS). It allows users to easily set up, operate, and scale a data warehouse in the cloud.

Amazon Web Services 221

Amazon Web Services AWS Data Warehouse Cloud 221

Christophe Blefari

MARCH 15, 2024

Differential storage, a building block for a DuckDB-based data warehouse — It's MotherDuck vision, creating the next data warehouse on-top of DuckDB leveraging DuckDB morphing capacities between a single machine and a production ecosystem.

Metadata 272

Metadata Data Datasets Data Warehouse 272

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessible 162

Confessions of a Data Guy

MAY 30, 2024

Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending.

Building 184

Building Data Warehouse Data Data Engineering 184

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Hevo

DECEMBER 6, 2024

This is why we talk about a marketing data warehouse today that is powerful enough to make […] The variety of campaigns running through platforms, such as Google Ads, Facebook, and HubSpot, among others, gives the kind of information that would just flood you.

Data Warehouse 52

Data Warehouse Data IT 52

Christophe Blefari

JUNE 21, 2024

Snowflake was founded in 2012 around its data warehouse product, which is still its core offering, and Databricks was founded in 2013 from academia with Spark co-creator researchers, becoming Apache Spark in 2014. Databricks is focusing on simplification (serverless, auto BI 2 , improved PySpark) while evolving into a data warehouse.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Data Engineering Podcast

APRIL 9, 2023

When it was difficult to wire together the event collection, data modeling, reporting, and activation it made sense to buy monolithic products that handled every stage of the customer data lifecycle. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Confessions of a Data Guy

JANUARY 12, 2023

Anyone who’s been roaming around the forest of Data Engineering has probably run into many of the newish tools that have been growing rapidly around the concepts of Data Warehouses, Data Lakes, and Lake Houses … the merging of the old relational database functionality with TB and PB level cloud-based file storage systems.

Data Lake 162

Data Lake Relational Database Data Warehouse Data Engineering 162

Data Engineering Podcast

JUNE 25, 2023

You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. Go to [dataengineeringpodcast.com/materialize]([link] Support Data Engineering Podcast

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Start Data Engineering

OCTOBER 25, 2023

Six Steps for a Clean Data Warehouse 2.1. Make data easy to use with the appropriate data model 2.3. Good input data is necessary for a good data warehouse 2.4. Introduction 2. Understand the business 2.2. Define Source of Truth (SOT) and trace its usage 2.5. Watch out for org-level red flags ?

Data Warehouse 130

Data Warehouse Data IT 130

databricks

OCTOBER 14, 2024

Over the last few years, we've seen tremendous growth and adoption of Databricks SQL , our intelligent data warehouse purpose-built on the Data.

SQL 131

SQL Data Warehouse Data 131

databricks

JANUARY 31, 2024

Welcome to the blog series covering product advancements in 2023 for Databricks SQL, the serverless data warehouse from Databricks. This is part 2.

SQL 130

SQL Programming Data Warehouse Data 130

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Seattle Data Guy

MAY 7, 2024

Getting data out of source systems and into a data warehouse or data lake is one of the first steps in making it usable by analysts and data scientists. The question is how will your team do that?

Data Lake 130

Data Lake Data Warehouse Data Systems 130

Seattle Data Guy

MAY 31, 2024

How companies data model varies widely. However, when you look in their data warehouse the only part you recognize is the word fact and dim. Over the past near decade, I have worked for and with different companies that have used various methods to capture this data.…

Data Warehouse 130

Data Warehouse Data 130

Christophe Blefari

DECEMBER 30, 2022

In a nutshell data modeling is the techniques we can use to structure the data in data warehouses. We often use the Snowflake Schema or the Star Schema (that is a special case of the previous one, here Snowflake is not the data warehouse technology but more the shape of the table relationships—drawing a snowflake).

Data Warehouse 130

Data Warehouse BI Data Machine Learning 130

Data Engineering Podcast

JUNE 4, 2023

How does the unified experience of Agile Data Engine change the way that teams think about the lifecycle of their data? What does CI/CD look like for a data warehouse? Can you describe how Agile Data Engine is architected? What does CI/CD look like for a data warehouse?

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

databricks

OCTOBER 23, 2024

Migrating your data warehouse workloads is one of the most challenging yet essential tasks for any organization. Whether the motivation is the growth.

Data Warehouse 116

Data Warehouse Data 116