phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Grouparoo

JANUARY 11, 2022

This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle. There are two main options available, a data lake and a data warehouse. What is a Data Warehouse? What is a Data Lake?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Though basic and easy to use, traditional table storage formats struggle to keep up. Track data files within the table along with their column statistics.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Knowledge Hut

APRIL 23, 2024

Two popular approaches that have emerged in recent years are data warehouse and big data. While both deal with large datasets, but when it comes to data warehouse vs big data, they have different focuses and offer distinct advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Monte Carlo

OCTOBER 31, 2024

There are dozens of data engineering tools available on the market, so familiarity with a wide variety of these can increase your attractiveness as an AI data engineering candidate. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Christophe Blefari

JUNE 21, 2024

Snowflake was founded in 2012 around its data warehouse product, which is still its core offering, and Databricks was founded in 2013 from academia with Spark co-creator researchers, becoming Apache Spark in 2014. Databricks is focusing on simplification (serverless, auto BI 2 , improved PySpark) while evolving into a data warehouse.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Sync Computing

NOVEMBER 7, 2024

A brief history of data storage The value of data has been apparent for as long as people have been writing things down. Data volume and velocity, governance, structure, and regulatory requirements have all evolved and continue to. The data warehouse concept dates back to data marts in the 1970s.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

Monte Carlo

AUGUST 25, 2023

That’s why it’s essential for teams to choose the right architecture for the storage layer of their data stack. But, the options for data storage are evolving quickly. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Christophe Blefari

NOVEMBER 11, 2022

I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Kovid wrote an article that tries to explain what are the ingredients of a data warehouse. And he does it well. In the post Kovid details every idea.

BI 130

BI Data Warehouse Data Database 130

Data Engineering Weekly

FEBRUARY 2, 2025

[link] Get Your Guide: From Snowflake to Databricks: Our cost-effective journey to a unified data warehouse. GetYourGuide discusses migrating its Business Intelligence (BI) data source from Snowflake to Databricks, achieving a 20% cost reduction.

Data Engineering 58

Data Engineering Data Engineer Engineering Data Lake 58

Knowledge Hut

APRIL 25, 2024

Each of these technologies has its own strengths and weaknesses, but all of them can be used to gain insights from large data sets. As organizations continue to generate more and more data, big data technologies will become increasingly essential. Let's explore the technologies available for big data.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Cloudyard

JANUARY 21, 2025

In this blog, we’ll explore the significance of schema evolution using real-world examples with CSV, Parquet, and JSON data formats. Schema evolution allows for the automatic adjustment of the schema in the data warehouse as new data is ingested, ensuring data integrity and avoiding pipeline failures.

Data Schemas 59

Data Schemas Data Pipeline Data Warehouse Data Storage 59

Striim

SEPTEMBER 11, 2024

Striim, for instance, facilitates the seamless integration of real-time streaming data from various sources, ensuring that it is continuously captured and delivered to big data storage targets. This method is advantageous when dealing with structured data that requires pre-processing before storage.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Data Engineering Weekly

OCTOBER 11, 2024

This centralized model mirrors early monolithic data warehouse systems like Teradata, Oracle Exadata, and IBM Netezza. These systems provided centralized data storage and processing at the cost of agility. Data engineering followed a similar path.

Manufacturing 110

Manufacturing Transportation Data Warehouse Unstructured Data 110

Christophe Blefari

SEPTEMBER 25, 2023

A guide to the Snowflake results cache — Cache is a critical piece to every data warehouse either for reusing data between runs or between stages in the same run. I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines.

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

SEPTEMBER 25, 2023

A guide to the Snowflake results cache — Cache is a critical piece to every data warehouse either for reusing data between runs or between stages in the same run. I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines.

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Picking the right format for your data storage.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Monte Carlo

NOVEMBER 21, 2024

This approach is fantastic when you’re not quite sure how you’ll need to use the data later, or when different teams might need to transform it in different ways. It’s more flexible than ETL and works great with the low cost of modern data storage.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Cloudera

DECEMBER 12, 2018

After having rebuilt their data warehouse, I decided to take a little bit more of a pointed role, and I joined Oracle as a database performance engineer. I spent eight years in the real-world performance group where I specialized in high visibility and high impact data warehousing competes and benchmarks.

Data Warehouse 42

Data Warehouse Relational Database Hadoop Database 42

Monte Carlo

JANUARY 25, 2023

When it comes to the question of building or buying your data stack, there’s never a one-size-fits-all solution for every data team—or every component of your data stack. Data storage and compute are very much the foundation of your data platform. Let’s jump in!

Data Warehouse 40

Data Warehouse Building Data Lake Data Storage 40

Cloudera

MAY 30, 2024

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission-critical, large-scale data analytics and AI use cases—including enterprise data warehouses. This scalability ensures the data lakehouse remains responsive and performant, even as data complexity and usage patterns change over time.

Data Lake 74

Data Lake Data Warehouse Programming Language Data Ingestion 74

Cloudera

JANUARY 21, 2021

While cloud-native, point-solution data warehouse services may serve your immediate business needs, there are dangers to the corporation as a whole when you do your own IT this way. Cloudera Data Warehouse (CDW) is here to save the day! CDW is an integrated data warehouse service within Cloudera Data Platform (CDP).

IT 94

IT Data Lake Data Warehouse Cloud Storage 94

Ascend.io

NOVEMBER 14, 2024

Snowflake and Azure Synapse offer powerful data warehousing solutions that simplify data integration and analysis by providing elastic scaling and optimized query performance. These techniques minimize the amount of data that needs to be processed at any given time, leading to significant cost savings.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

DECEMBER 16, 2019

You work hard to make sure that your data is clean, reliable, and reproducible throughout the ingestion pipeline, but what happens when it gets to the data warehouse? Dataform picks up where your ETL jobs leave off, turning raw data into reliable analytics.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Weekly

JUNE 10, 2024

[link] Piethein Strengholt: Integrating Azure Databricks and Microsoft Fabric Databricks buying Tabluar certainly triggers interesting patterns in the data infrastructure. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure. Will they co-exist or fight with each other?

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

Monte Carlo

MARCH 3, 2025

Prior to data powering valuable data products like machine learning models and real-time marketing applications, data warehouses were mainly used to create charts in binders that sat off to the side of board meetings. This pattern is repeating with AI. 2023 was the year of GPUs. 2024 was the year of foundational models.

Unstructured Data 52

Unstructured Data Data Cloud Computing Banking 52

Towards Data Science

DECEMBER 1, 2023

This tool automates ELT (Extract, Load, Transform) process, integrating your data from the source system of Google Calendar to our Snowflake data warehouse. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 97

Data Engineering Data Engineer Project Engineering 97

Hevo

SEPTEMBER 11, 2024

Nowadays, the term is used for petabytes or even exabytes of data (1024 Petabytes), close to trillions of records from billions of people. In this fast-moving landscape, the key to making a difference is picking up the correct data storage solution for your business. […]

Data Lake 40

Data Lake Data Warehouse Database Data Storage 40

Hevo

JULY 23, 2024

With the vast amount of data being collected today for various purposes, there is an increasing need to find the proper data storage, which also heavily depends on your specific analytical objectives. This […]

Data Lake 40

Data Lake Data Warehouse Data Storage Data 40

Towards Data Science

FEBRUARY 15, 2023

Concepts, theory, and functionalities of this modern data storage framework Photo by Nick Fewings on Unsplash Introduction I think it’s now perfectly clear to everybody the value data can have. To use a hyped example, models like ChatGPT could only be built on a huge mountain of data, produced and collected over years.

Data Lake 98

Data Lake Data Warehouse Hadoop Architecture 98

Data Engineering Podcast

AUGUST 14, 2021

Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. No more scripts, just SQL.

Unstructured Data 100

Unstructured Data Machine Learning Data Lake SQL 100

Data Engineering Weekly

DECEMBER 2, 2023

Revisiting The Current State of Data Infrastructure Let’s revisit the current state of the data infrastructure before discussing the S3 Express. There are two critical properties of data warehouse access patterns. Data freshness matters a lot—the more recent the data, the more frequently it is accessed.

IT 85

IT BI AWS Kafka 85