Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs.

Data Lake 114

Data Lake Data Warehouse Cloud Unstructured Data 114

Snowflake

SEPTEMBER 14, 2023

More than 50% of data leaders recently surveyed by BCG said the complexity of their data architecture is a significant pain point in their enterprise. As a result,” says BCG, “many companies find themselves at a tipping point, at risk of drowning in a deluge of data, overburdened with complexity and costs.”

Data Architecture 120

Data Architecture Architecture Hadoop Data Warehouse 120

Christophe Blefari

JANUARY 11, 2025

They developed a /data command internally that answer questions about everything and structured the analytics around a foundational data platform with company-wide analytics data layer that provides time series efficiency metrics across various business use cases. when you have a semantic layer).

Data 130

Data Data Warehouse Coding Programming Language 130

François Nguyen

MARCH 22, 2021

With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. We are Data Teams versus we have to patch the server with the latest version and do the tests.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). To start, can you share your definition of what constitutes a "Data Lakehouse"? Want to see Starburst in action?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

JULY 30, 2024

Rethinking data warehousing: Why redefinition is necessary even beyond Modern Data Warehouse (MDW) and Lakehouse Models Continue reading on Towards Data Science »

Data Warehouse 75

Data Warehouse Data Science Data Data Architecture 75

Cloudera

MAY 28, 2024

It’s not enough for businesses to implement and maintain a data architecture. The unpredictability of market shifts and the evolving use of new technologies means businesses need more data they can trust than ever to stay agile and make the right decisions.

Data Architecture 88

Data Architecture Architecture Data Lake Data Warehouse 88

Cloudera

JULY 11, 2022

What used to be bespoke and complex enterprise data integration has evolved into a modern data architecture that orchestrates all the disparate data sources intelligently and securely, even in a self-service manner: a data fabric. Cloudera data fabric and analyst acclaim. Next steps.

Data Architecture 107

Data Architecture Architecture Data Government 107

Confluent

OCTOBER 30, 2020

A fundamental challenge with today’s “data explosion” is finding the best answer to the question, “So where do I put my data?” while avoiding the longer-term problem of data warehouses, […].

Data Architecture 119

Data Architecture Architecture Kafka Building 119

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 102

Architecture Metadata Machine Learning Unstructured Data 102

Cloudera

MARCH 2, 2022

Folding data into pointers. On 64-bit architectures, pointers store memory addresses using 8 bytes. But on architectures like x86 and ARM the linear address is limited to 48 bits long, with bits 49 to 64 reserved for future usage. Intel Level 5 proposal 64-bit memory address.

Data Warehouse 89

Data Warehouse Bytes Data Business Intelligence 89

Analytics Vidhya

FEBRUARY 1, 2023

Introduction Nowadays, organizations are looking for multiple solutions to deal with big data and related challenges.

Data Cleanse 223

Data Cleanse Data Warehouse Big Data Cloud 223

Data Engineering Podcast

AUGUST 27, 2021

By making the software be the owner of the data that it generates, we have to go through the trouble of extracting the information to then be used elsewhere. The team at Cinchy are working to bring about a new paradigm of software architecture that puts the data as the central element. No more scripts, just SQL.

Data Integration 100

Data Integration Architecture Data Warehouse Data Lake 100

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB Data Warehouse 100

Hevo

SEPTEMBER 10, 2024

Nowadays, when it comes to data management, every business has to make one critical decision: whether to use a Data Mesh or a Data Warehouse. Both are strong data management architectures, but they are designed to support different needs and various organizational structures.

Data Warehouse 52

Data Warehouse Architecture Data Architecture Data 52

Cloudera

AUGUST 30, 2022

Modern data architectures. To eliminate or integrate these silos, the public sector needs to adopt robust data management solutions that support modern data architectures (MDAs). Deploying modern data architectures. Lack of sharing hinders the elimination of fraud, waste, and abuse. Forrester ).

Data Architecture 79

Data Architecture Architecture Data Lake NoSQL 79

Monte Carlo

JANUARY 14, 2025

Our calculator estimates the cost of this poor data quality would be: 400 data incidents per year 2400 data downtime hours per year $156,587 in resource cost $2,671,232 in efficiency cost The Data Quality Calculator provides the estimated cost of bad data by leveraging data from hundreds of data warehouses and millions of tables.

Data 98

Data Data Engineer Data Engineering Media 98

Sync Computing

NOVEMBER 7, 2024

Data volume and velocity, governance, structure, and regulatory requirements have all evolved and continue to. Despite these limitations, data warehouses, introduced in the late 1980s based on ideas developed even earlier, remain in widespread use today for certain business intelligence and data analysis applications.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. How has that changed the architectural approach to CDPs? Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. Go to [dataengineeringpodcast.com/materialize]([link] Support Data Engineering Podcast

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Striim

MARCH 21, 2025

With instant elasticity, high-performance, and secure data sharing across multiple clouds , Snowflake has become highly in-demand for its cloud-based data warehouse offering. As organizations adopt Snowflake for business-critical workloads, they also need to look for a modern data integration approach.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Engineering at Meta

FEBRUARY 4, 2025

Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand.

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

Striim

MARCH 21, 2025

Batch processing: data is typically extracted from databases at the end of the day, saved to disk for transformation, and then loaded in batch to a data warehouse. Batch data integration is useful for data that isn’t extremely time-sensitive. Electric bills are a relevant example.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Data Engineering Podcast

FEBRUARY 4, 2024

Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. What are some of the platforms/architectures that teams are replacing with RisingWave? Want to see Starburst in action?

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Data Engineering Weekly

AUGUST 6, 2024

CDC tools fuel analytical apps and mission-critical data feeds in banking and regulated industries, with use cases ranging from data synchronization, managing risk, and preventing fraud to driving personalization. This approach simplifies data architecture and enhances performance by reducing data movement and latency.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

JUNE 9, 2024

Data observability has been gaining adoption for a number of years now, with a large focus on data warehouses. How much of the complexity is due to the nature of streaming data vs. the architectural realities of Flink? How have the requirements of generative AI shifted the demand for streaming data systems?

Process 147

Process Data Lake High Quality Data Machine Learning 147

phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Data Engineering Weekly

FEBRUARY 2, 2025

[link] Adam Bellemare & Thomas Betts: The End of the Bronze Age: Rethinking the Medallion Architecture I’m always a bit uncomfortable with medallion architecture since it is a glorified term for the traditional ETL process. link] All rights reserved ProtoGrowth Inc, India.

Data Engineer 57

Data Engineer Data Engineering Engineering Data Lake 57

Data Engineering Podcast

MAY 21, 2023

Can you describe the architecture of your Flow platform? What is involved in getting Flow/Estuary deployed and integrated with an organization's data systems? How does it impact the overall system architecture for a data platform as compared to other prevalent paradigms? RudderStack also supports real-time use cases.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Towards Data Science

JANUARY 16, 2024

My personal take on justifying the existence of Data Mesh A senior stakeholder at one my projects mentioned that they wanted to decentralise their data platform architecture and democratise data across the organisation. When I heard the words ‘decentralised data architecture’, I was left utterly confused at first!

Data Warehouse 95

Data Warehouse Data Architecture Relational Database NoSQL 95

Striim

AUGUST 31, 2023

Kappa Architectures are becoming a popular way of unifying real-time (streaming) and historical (batch) analytics giving you a faster path to realizing business value with your pipelines. Kappa Architecture combines streaming and batch while simultaneously turning data warehouses and data lakes into near real-time sources of truth.

Data Integration 69

Data Integration Architecture Amazon Web Services Machine Learning 69

Monte Carlo

NOVEMBER 21, 2024

Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way. That’s where data pipeline design patterns come in. Lambda Architecture Pattern 4.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Podcast

JUNE 4, 2023

What are some of the tools and architectures that an organization might be able to replace with Agile Data Engine? How does the unified experience of Agile Data Engine change the way that teams think about the lifecycle of their data? What does CI/CD look like for a data warehouse?

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Monte Carlo

NOVEMBER 22, 2024

Let’s walk through how to transform your scrappy data setup into a robust pipeline that’s ready to grow with your business. Gone are the days of just dumping everything into a single database; modern data architectures typically use a combination of data lakes and warehouses.

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Data Engineering Podcast

MAY 14, 2023

In this episode the host Tobias Macey shares his reflections on recent experiences where the abstractions leaked and some observances on how to deal with that situation in a data platform architecture. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Lake 147

Data Lake Machine Learning Data Warehouse AWS 147