Monte Carlo

JANUARY 25, 2023

When it comes to the question of building or buying your data stack, there’s never a one-size-fits-all solution for every data team—or every component of your data stack. Data storage and compute are very much the foundation of your data platform. Let’s jump in!

Data Warehouse 40

Data Warehouse Building Data Lake Data Storage 40

Hevo

JULY 23, 2024

With the vast amount of data being collected today for various purposes, there is an increasing need to find the proper data storage, which also heavily depends on your specific analytical objectives. This […]

Data Lake 40

Data Lake Data Warehouse Data Storage Data 40

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Christophe Blefari

JUNE 21, 2024

Snowflake was founded in 2012 around its data warehouse product, which is still its core offering, and Databricks was founded in 2013 from academia with Spark co-creator researchers, becoming Apache Spark in 2014. Databricks is focusing on simplification (serverless, auto BI 2 , improved PySpark) while evolving into a data warehouse.

Metadata 147

Metadata Data Warehouse BI Scala 147

Striim

SEPTEMBER 11, 2024

Striim, for instance, facilitates the seamless integration of real-time streaming data from various sources, ensuring that it is continuously captured and delivered to big data storage targets. This method is advantageous when dealing with structured data that requires pre-processing before storage.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Towards Data Science

MARCH 23, 2023

The difference between tables and views and how to use them Continue reading on Towards Data Science »

Data Warehouse 74

Data Warehouse Data Science Data Cloud Computing 74

Data Engineering Weekly

OCTOBER 11, 2024

This centralized model mirrors early monolithic data warehouse systems like Teradata, Oracle Exadata, and IBM Netezza. These systems provided centralized data storage and processing at the cost of agility. Data engineering followed a similar path.

Manufacturing 109

Manufacturing Transportation Data Warehouse Unstructured Data 109

Hevo

JUNE 25, 2024

Organizations face a discernible lag in performance with the ever-increasing rise in data. Traditional data warehouses become a financial burden with time despite proper planning as companies also suffer storage limitations.

Data Warehouse 52

Data Warehouse Data Storage Cloud Process 52

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Though basic and easy to use, traditional table storage formats struggle to keep up. Track data files within the table along with their column statistics.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Hevo

AUGUST 2, 2024

In the modern data-centric world, efficient data transfer and management are essential to staying competitive. Most businesses use a data warehouse for their data storage solution, and one of the leading data warehousing solutions is Amazon Redshift. In 2024, over 11441 companies1 […]

AWS 52

AWS Database-centric Data Warehouse Data Storage 52

Knowledge Hut

APRIL 25, 2024

Each of these technologies has its own strengths and weaknesses, but all of them can be used to gain insights from large data sets. As organizations continue to generate more and more data, big data technologies will become increasingly essential. Let's explore the technologies available for big data.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Cloudera

DECEMBER 12, 2018

After having rebuilt their data warehouse, I decided to take a little bit more of a pointed role, and I joined Oracle as a database performance engineer. I spent eight years in the real-world performance group where I specialized in high visibility and high impact data warehousing competes and benchmarks.

Data Warehouse 42

Data Warehouse Relational Database Hadoop Database 42

Simon Späti

NOVEMBER 28, 2018

However, this is still not common in the Data Warehouse (DWH) field. In my recent blog, I researched OLAP technologies, for this post I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system. Why is this?

Data Warehouse 130

Data Warehouse Data Storage Data Architecture Architecture 130

Hevo

MAY 24, 2024

For decades, traditional On-Premise Data Warehouses have been tightly coupled with Data Storage and Computing, making them difficult to scale.

Unstructured Data 52

Unstructured Data Data Warehouse Data Storage Data 52

Hevo

MAY 10, 2024

Amazon S3 is a prominent data storage platform with multiple storage and security features. Integrating data stored in Amazon S3 to a data warehouse like Databricks can enable better data-driven decisions. Integrating data from Amazon S3 to Databricks […]

Data Integration 52

Data Integration Data Warehouse Data Storage Machine Learning 52

Christophe Blefari

NOVEMBER 11, 2022

I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Kovid wrote an article that tries to explain what are the ingredients of a data warehouse. And he does it well. In the post Kovid details every idea.

BI 130

BI Data Warehouse Data Database 130

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Picking the right format for your data storage.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Does not have the resources to implement robust data governance and management.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Does not have the resources to implement robust data governance and management.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Does not have the resources to implement robust data governance and management.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

Databand.ai

JULY 19, 2023

A Beginner’s Guide [SQ] Niv Sluzki July 19, 2023 ELT is a data processing method that involves extracting data from its source, loading it into a database or data warehouse, and then later transforming it into a format that suits business needs. The data is loaded as-is, without any transformation.

Data Cleanse 52

Data Cleanse Raw Data Data Storage Data Warehouse 52

Christophe Blefari

SEPTEMBER 25, 2023

A guide to the Snowflake results cache — Cache is a critical piece to every data warehouse either for reusing data between runs or between stages in the same run. I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

A guide to the Snowflake results cache — Cache is a critical piece to every data warehouse either for reusing data between runs or between stages in the same run. I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

FEBRUARY 24, 2023

In order to improve your data infra you should sometimes try to occasionally kill your data stack , chaos engineering is something that helps discover issues. At the moment the best way to develop in your data warehouse is still in the query editor of BigQuery / Snowflake / etc., Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130

Hevo

MAY 18, 2023

A plain text file containing a list of data separated by commas is known as a Comma Separated Values (CSV) file. Data is regularly transferred between apps using these files. BigQuery, Google’s data warehouse as a service, combines data storage and analytics in one package. BigQuery allows […]

Data Warehouse 52

Data Warehouse Data Storage Data Database 52

Data Engineering Weekly

OCTOBER 6, 2024

The key benefits are Improved data quality, Enhanced data governance Increased security Cost efficiency [link] HomeToGo: How HomeToGo improved our Superset Monitoring Framework Apache Superset is the most popular open-source BI tool in the industry. “text-to-SQL” and “text-to-insight.”

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Monte Carlo

APRIL 27, 2023

Empowered by the rise of the modern data stack , leading companies like Warner Music Group , Chime and Petsmart are now turning to the Composable CDP to unlock a modular customer data architecture that centers on the cloud data warehouse. Why not build one single store of customer data to activate from centrally?

Data Warehouse 72

Data Warehouse Data Collection Architecture Data Storage 72

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. What is a Data Lake? What are Data Modeling Methodologies, and Why Are They Important for a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

AUGUST 21, 2024

The Awards showcase IT vendor offerings that provide significant technology advances – and partner growth opportunities – across technology categories including AI and AI infrastructure, cloud management tools, IT infrastructure and monitoring, networking, data storage, and cybersecurity.

Unstructured Data 65

Unstructured Data Business Intelligence Data Architecture Data Warehouse 65

Knowledge Hut

MARCH 28, 2024

Job Role 1: Azure Data Engineer Azure Data Engineers develop, deploy, and manage data solutions with Microsoft Azure data services. They use many data storage, computation, and analytics technologies to develop scalable and robust data pipelines.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

OCTOBER 31, 2024

There are dozens of data engineering tools available on the market, so familiarity with a wide variety of these can increase your attractiveness as an AI data engineering candidate. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Hevo

MAY 17, 2024

You can use it effectively for customer data management, but it does not facilitate efficient data storage. To resolve this limitation, you can integrate Salesforce with data warehouses like Snowflake. Snowflake It allows you […]

Cloud 40

Cloud Media Data Warehouse Data Storage 40

Hevo

MAY 9, 2024

Nowadays, with ever-increasing data, storage and analysis of large volumes of data have become basic foundations of any organization. While focusing on storage and analysis, which two concepts come to your mind? That’s a database and a data warehouse, right?

AWS 40

AWS Data Warehouse Data Storage Database 40

Hevo

AUGUST 17, 2023

Organizations rely on multiple databases and cloud-based solutions for their ever-expanding data storage needs. However, for in-depth analysis of organizational data, it is essential to perform seamless integrations between different platforms.

MySQL 52

MySQL Data Warehouse Data Storage Database 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 81

Data Warehouse Database-centric Metadata Cloud 81

DareData

JANUARY 30, 2023

In this post, we'll discuss some key data engineering concepts that data scientists should be familiar with, in order to be more effective in their roles. These concepts include concepts like data pipelines, data storage and retrieval, data orchestrators or infrastructure-as-code.

Data Engineering 52

Data Engineering Data Engineer NoSQL Engineering 52