Towards Data Science

JULY 30, 2024

Rethinking data warehousing: Why redefinition is necessary even beyond Modern Data Warehouse (MDW) and Lakehouse Models Continue reading on Towards Data Science »

Data Warehouse 75

Data Warehouse Data Science Data Data Architecture 75

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs. Want to see these features in action?

Data Lake 107

Data Lake Data Warehouse Cloud Unstructured Data 107

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Data Warehouse 100

Data Warehouse MySQL Hadoop Data Lake 100

Confluent

OCTOBER 30, 2020

A fundamental challenge with today’s “data explosion” is finding the best answer to the question, “So where do I put my data?” while avoiding the longer-term problem of data warehouses, […].

Data Architecture 120

Data Architecture Architecture Kafka Building 120

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 79

Data Lake Cloud Storage Metadata Data Warehouse 79

Cloudera

AUGUST 30, 2022

Modern data architectures. To eliminate or integrate these silos, the public sector needs to adopt robust data management solutions that support modern data architectures (MDAs). Deploying modern data architectures. Lack of sharing hinders the elimination of fraud, waste, and abuse. Forrester ).

Data Architecture 79

Data Architecture Architecture Data Lake NoSQL 79

Monte Carlo

JANUARY 14, 2025

Our calculator estimates the cost of this poor data quality would be: 400 data incidents per year 2400 data downtime hours per year $156,587 in resource cost $2,671,232 in efficiency cost The Data Quality Calculator provides the estimated cost of bad data by leveraging data from hundreds of data warehouses and millions of tables.

Data 98

Data Data Engineering Data Engineer Media 98

Hevo

SEPTEMBER 10, 2024

Nowadays, when it comes to data management, every business has to make one critical decision: whether to use a Data Mesh or a Data Warehouse. Both are strong data management architectures, but they are designed to support different needs and various organizational structures.

Data Warehouse 52

Data Warehouse Architecture Data Architecture Data 52

Simon Späti

NOVEMBER 28, 2018

However, this is still not common in the Data Warehouse (DWH) field. In my recent blog, I researched OLAP technologies, for this post I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system. These days, everyone talks about open-source.

Data Warehouse 130

Data Warehouse Data Storage Data Architecture Architecture 130

Cloudera

SEPTEMBER 29, 2022

Each of these trends claim to be complete models for their data architectures to solve the “everything everywhere all at once” problem. Data teams are confused as to whether they should get on the bandwagon of just one of these trends or pick a combination. First, we describe how data mesh and data fabric could be related.

Architecture 74

Architecture Data Architecture Metadata Data Warehouse 74

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Data Engineering Weekly

AUGUST 6, 2024

CDC tools fuel analytical apps and mission-critical data feeds in banking and regulated industries, with use cases ranging from data synchronization, managing risk, and preventing fraud to driving personalization. This approach simplifies data architecture and enhances performance by reducing data movement and latency.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Towards Data Science

JANUARY 16, 2024

When I heard the words ‘decentralised data architecture’, I was left utterly confused at first! In my then limited experience as a Data Engineer, I had only come across centralised data architectures and they seemed to be working very well. Result: Data warehouse was born. So what was missing?

Data Warehouse 95

Data Warehouse Data Architecture Relational Database NoSQL 95

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Acceldata

DECEMBER 19, 2022

Avoid these three data pitfalls when attempting to modernize your data architecture with a data lake or data warehouse.

Data Lake 52

Data Lake Data Warehouse Data Architecture Data 52

Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Data Architect 52

Data Engineering Podcast

JUNE 17, 2023

Sign up free at dataengineeringpodcast.com/rudderstack - Your host is Tobias Macey and today I'm interviewing Satish Jayanthi about the practice and promise of building a column-aware data architecture through intentional modeling Interview Introduction How did you get involved in the area of data management?

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

phData: Data Engineering

NOVEMBER 8, 2024

Versioning also ensures a safer experimentation environment, where data scientists can test new models or hypotheses on historical data snapshots without impacting live data. Note : Cloud Data warehouses like Snowflake and Big Query already have a default time travel feature. FAQs What is a Data Lakehouse?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Snowflake

JULY 27, 2023

Over the course of this journey, HomeToGo’s data needs have evolved considerably. It also came with other advantages such as independence of cloud infrastructure providers, data recovery features such as Time Travel , and zero copy cloning which made setting up several environments — such as dev, stage or production — way more efficient.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to reduce repetitive work for common workflows by adopting a visual pipeline builder to support your data warehouse transformations.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. The past decades of enterprise data platform architectures can be summarized in 69 words. Introduction to Data Mesh. Source: Thoughtworks.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Architecture 98

Monte Carlo

NOVEMBER 22, 2024

Gone are the days of just dumping everything into a single database; modern data architectures typically use a combination of data lakes and warehouses. Think of your data lake as a vast reservoir where you store raw data in its original form—great for when you’re not quite sure how you’ll use it yet.

Data Engineering 52

Data Engineering Data Engineer Building Engineering 52

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Hevo

SEPTEMBER 17, 2024

Organizations have begun to built data warehouses and lakes to analyze large amounts of data for insights and business reports. Often time they bring data from multiple data silos into their data lake and also have data stored in particular data stores like NoSQL databases to support different use cases.

Data Architecture 40

Data Architecture Architecture NoSQL Data Lake 40

Data Engineering Podcast

JULY 3, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Cloudera

JUNE 18, 2022

These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake.

Data Lake 94

Data Lake Data Warehouse BI SQL 94

Snowflake

FEBRUARY 21, 2023

In this Q&A, we hear from Nico Acosta, CEO and Co-Founder of Propel, about how his company is building an API platform to equip developers to build with data, and why data architecture is the most important technical decision a company will make. We fully align with Snowflake’s vision of the Data Cloud.

AWS 131

AWS Building Raw Data Architecture 131

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

Data Engineering Podcast

JUNE 16, 2019

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. The Lakehouse architecture was one of them.

Data Lake 98

Data Lake Data Warehouse Hadoop Architecture 98

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. The data you’re looking for is already in your data warehouse and BI tools.

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

OCTOBER 2, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. What are the driving factors for building a real-time data platform?

Building 100

Building BI Data Architecture Architecture 100

Snowflake

JANUARY 30, 2023

By running data warehouse and data engineering workloads on Snowflake’s Data Cloud Ramp improves performance and user experience, while delivering powerful insights to customers quickly. How do you scale seamlessly, without worrying about keeping the lights on?

Data Architecture 52

Data Architecture Architecture Management Datasets 52

Cloudera

JUNE 30, 2022

Companies, on the other hand, have continued to demand highly scalable and flexible analytic engines and services on the data lake, without vendor lock-in. Organizations want modern data architectures that evolve at the speed of their business and we are happy to support them with the first open data lakehouse. .

Data Lake 88

Data Lake Business Intelligence Metadata Data Warehouse 88