Engineering at Meta

JANUARY 22, 2025

These stages propagate through various systems including function-based systems that load, process, and propagate data through stacks of function calls in different programming languages (e.g., For simplicity, we will demonstrate these for the web, the data warehouse, and AI, per the diagram below. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). For someone who is interested in building a data lakehouse with Trino and Iceberg, how does that influence their selection of other platform elements?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

APRIL 23, 2023

Applications powered by real-time data were the exclusive domain of large and/or sophisticated tech companies for several years due to the inherent complexities involved. What are the shifts that have made them more accessible to a wider variety of teams? RudderStack also supports real-time use cases.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Snowflake

DECEMBER 4, 2024

The next evolution in data is making it AI ready. For years, an essential tenet of digital transformation has been to make data accessible, to break down silos so that the enterprise can draw value from all of its data. For this reason, internal-facing AI will continue to be the focus for the next couple of years.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessibility 162

phData: Data Engineering

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Snowflake

NOVEMBER 11, 2024

This is not surprising when you consider all the benefits, such as reducing complexity [and] costs and enabling zero-copy data access (ideal for centralizing data governance).

Banking 98

Banking Finance Retail Food 98

Netflix Tech

DECEMBER 19, 2024

For convenience, they support the dot-syntax (when possible) for accessing keys, making it easy to access values in a nested configuration. You can access Configs of any past runs easily through the Client API. This has been a guiding design principle with Metaflow since its inception. nflxfastdata(2.13.5);nflx(2.13.5);metaboost(0.0.27)

Machine Learning 81

Machine Learning Project Data Warehouse Coding 81

Sync Computing

NOVEMBER 7, 2024

While a +3500 year data retention capability for data stored on clay tablets is impressive, the access latency and forward compatibility of clay tablets fall a little short. Similarly, data platforms based on the business needs of the past dont always meet the needs of today. Save up to 50% on compute! Book a Demo!

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Monte Carlo

SEPTEMBER 10, 2024

This truth was hammered home recently when ride-hailing giant Uber found itself on the receiving end of a staggering €290 million ($324 million) fine from the Dutch Data Protection Authority. Poor data warehouse governance practices that led to the improper handling of sensitive European driver data. The reason?

Data Warehouse 52

Data Warehouse Government Data Governance Metadata 52

Edureka

APRIL 22, 2025

Fabric is meant for organizations looking for a single pane of glass across their data estate with seamless integration and a low learning curve for Microsoft users. Snowflake is a cloud-native platform for data warehouses that prioritizes collaboration, scalability, and performance. Office 365, Power BI, Azure).

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Hevo

OCTOBER 2, 2024

Data warehouses have transformed how companies store and manage data. By centralizing data into a single repository, overall data accessibility and quality improve a lot. Building an […]

Data Warehouse 52

Data Warehouse Data Accessibility Accessible 52

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. Go to [dataengineeringpodcast.com/materialize]([link] Support Data Engineering Podcast

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Monte Carlo

AUGUST 6, 2024

Data warehouses are the centralized repositories that store and manage data from various sources. They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. Integration Layer : Where your data transformations and business logic are applied.

Data Warehouse 52

Data Warehouse Raw Data Machine Learning BI 52

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements.

Management 90

Management Government Cloud Unstructured Data 90

Striim

NOVEMBER 19, 2024

This fully managed service leverages Striim Cloud’s integration with the Microsoft Fabric stack for seamless data mirroring to Fabric Data Warehouse and Lake House. Microsoft Azure Fabric is an end-to-end analytics and data platform designed for enterprises that require a unified solution.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Engineering at Meta

APRIL 28, 2025

Challenge Approach Understanding at scale (lack of foundation) At Meta, we manage hundreds of data systems and millions of assets across our family of apps. Each product features its own distinct data model, physical schema, query language, and access patterns. Datasets provide a native API for creating data pipelines.

Metadata 67

Metadata Data Utilities Data Warehouse 67

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

phData: Data Engineering

NOVEMBER 8, 2024

Hence, the metadata files record schema and partition changes, enabling systems to process data with the correct schema and partition structure for each relevant historical dataset. Data Versioning and Time Travel Open Table Formats empower users with time travel capabilities, allowing them to access previous dataset versions.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

Maxime Beauchemin

AUGUST 28, 2017

Consensus seeking Whether you think that old-school data warehousing concepts are fading or not, the quest to achieve conformed dimensions and conformed metrics is as relevant as it ever was. The data warehouse needs to reflect the business, and the business should have clarity on how it thinks about analytics.

Data Engineer 174

Data Engineer Data Engineering Engineering Software Engineer 174

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineer 258

Data Engineer Data Engineering Engineering Data 258

Snowflake

APRIL 22, 2025

And 81% of early adopters say theyre aggressively increasing their investments in cloud-based data warehousing solutions over the next 12 months. Thats no surprise the cloud offers greater scalability, cost control and governance as well as access to the high-performance compute needed for gen AI initiatives.

IT 54

IT Unstructured Data Government Cloud 54

Data Engineering Weekly

FEBRUARY 2, 2025

The article advocates for a "shift left" approach to data processing, improving data accessibility, quality, and efficiency for operational and analytical use cases. link] Get Your Guide: From Snowflake to Databricks: Our cost-effective journey to a unified data warehouse.

Data Engineer 58

Data Engineer Data Engineering Engineering Data Lake 58

Data Engineering Weekly

AUGUST 6, 2024

As a result, lakehouses support more dynamic and flexible data architectures, catering to a broader range of analytics and operational workloads. For instance, in a fast-paced retail environment, lakehouses can ensure that inventory data remains up-to-date and accurate in the data warehouse, optimizing supply chain efficiency.

Data Lake 126

Data Lake Data Warehouse Database Data Architecture 126

Data Engineering Podcast

APRIL 16, 2023

Summary Business intellingence has been chasing the promise of self-serve data for decades. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

JANUARY 8, 2023

With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL. Go to dataengineeringpodcast.com/materialize today and sign up for early access to get started.

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Data Engineering Podcast

JANUARY 15, 2023

With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL. Go to dataengineeringpodcast.com/materialize today and sign up for early access to get started.

Coding 130

Coding Building PostgreSQL Data Lake 130

Cloudera

DECEMBER 2, 2021

Cloudera secures your data by providing encryption at rest and in transit, multi-factor authentication, Single Sign On, robust authorization policies, and network security. It is part of the Cloudera Data Platform, or CDP , which runs on Azure and AWS, as well as in the private cloud. Firewall Exceptions for Network Egress.

PostgreSQL 123

PostgreSQL Data Lake Data Warehouse Retail 123

Christophe Blefari

NOVEMBER 11, 2022

I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Kovid wrote an article that tries to explain what are the ingredients of a data warehouse. And he does it well. The end-game dataset.

BI 130

BI Data Warehouse Data Database 130

Striim

MARCH 21, 2025

To access real-time data, organizations are turning to stream processing. There are two main data processing paradigms: batch processing and stream processing. Batch processing: data is typically extracted from databases at the end of the day, saved to disk for transformation, and then loaded in batch to a data warehouse.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Snowflake

SEPTEMBER 14, 2023

Unbound by the limitations of a legacy on-premises solution, its multi-cluster shared data architecture separates compute from storage, allowing data teams to easily scale up and down based on their needs. Data engineers spent 20% of their time on infrastructure issues such as tuning Spark jobs.

Data Architecture 122

Data Architecture Architecture Hadoop Data Warehouse 122

Data Engineering Podcast

FEBRUARY 4, 2024

What are the situations where RisingWave can/should be a system of record vs. a point-in-time view of data in transit, with a data warehouse/lakehouse as the longitudinal storage and query engine? What are the user/developer experience elements that you have prioritized most highly?

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Analytics Vidhya

FEBRUARY 27, 2023

Introduction Enterprises here and now catalyze vast quantities of data, which can be a high-end source of business intelligence and insight when used appropriately. Delta Lake allows businesses to access and break new data down in real time.

Data Lake 222

Data Lake Business Intelligence Designing Accessibility 222

Data Engineering Podcast

JUNE 9, 2024

Your host is Tobias Macey and today I'm interviewing Ronen Korman and Stav Elkayam about pulling back the curtain on your real-time data streams by bringing intuitive observability to Flink streams Interview Introduction How did you get involved in the area of data management? What are some of the unique challenges posed by Flink?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Picnic Engineering

MARCH 28, 2025

TL;DR Over the past decade, Picnic transformed its approach to dataevolving from a single, all-purpose data team into multiple specialized teams using a lean, scalable tech stack. We empowered analysts by giving them access and training to the same tools as engineers, dramatically increasing speed and impact. The impact was massive.

Data Warehouse 40

Data Warehouse PostgreSQL Python Machine Learning 40

Ascend.io

NOVEMBER 14, 2024

Snowflake and Azure Synapse offer powerful data warehousing solutions that simplify data integration and analysis by providing elastic scaling and optimized query performance. Encrypting data both at rest and in transit ensures that sensitive information remains protected from unauthorized access.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52