Engineering at Meta

APRIL 28, 2025

Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process. However, conducting these processes outside of developer workflows presented challenges in terms of accuracy and timeliness.

Metadata 83

Metadata Data Utilities Data Warehouse 83

Snowflake

JUNE 10, 2025

Instead, to save space, the column values are implied until materialized through a read query and only then are the values propagated through the metadata layer (Metadata.json → Snapshot → Manifest → Datafile → Row). Entire tables can be encrypted with a single key, or access can be controlled at the snapshot level.

Metadata 95

Metadata Software Engineering Software Engineer Technology 95

Netflix Tech

MAY 9, 2025

At Netflix, we embarked on a journey to build a robust event processing platform that not only meets the current demands but also scales for future needs. This blog post delves into the architectural evolution and technical decisions that underpin our Ads event processing pipeline.

Process 73

Process Building Metadata Kafka 73

Precisely

OCTOBER 31, 2024

While data products may have different definitions in different organizations, in general it is seen as data entity that contains data and metadata that has been curated for a specific business purpose. A data fabric weaves together different data management tools, metadata, and automation to create a seamless architecture.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Engineering at Meta

JANUARY 22, 2025

This belief has led us to developing Privacy Aware Infrastructure (PAI) , which offers efficient and reliable first-class privacy constructs embedded in Meta infrastructure to address different privacy requirements, such as purpose limitation , which restricts the purposes for which data can be processed and used. Hack, C++, Python, etc.)

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

Snowflake

NOVEMBER 12, 2024

Managing application state and metadata Use Hybrid Tables as the system of record for application configuration, user profiles, workflow state and other metadata that needs to be accessed with high concurrency. Customers such as Siemens and PowerSchool are leveraging Hybrid Tables to track state for a wide variety of use cases.

Food 97

Food Metadata Education Data Architect 97

ProjectPro

JUNE 6, 2025

For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. It offers a simple and efficient solution for data processing in organizations. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog.

AWS 66

AWS Scala Metadata Data Lake 66

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. Arroyo, a stream-processing platform, rebuilt their engine using DataFusion. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.

Metadata 272

Metadata Software Engineering Software Engineer Data Warehouse 272

Engineering at Meta

JANUARY 21, 2025

Strobelight is also not a single profiler but an orchestrator of many different profilers (even ad-hoc ones) that runs on all production hosts at Meta, collecting detailed information about CPU usage, memory allocations, and other performance metrics from running processes. Did someone say Metadata?

Technology 115

Technology Metadata Utilities Engineering 115

Cloudera

NOVEMBER 15, 2024

The manual process of switching between tools slows down their work, often leaving them reliant on rudimentary methods of keeping track of their findings. The metadata-driven approach ensures quick query planning so defenders don’t have to deal with slow processes when they need fast answers.

Metadata 84

Metadata Data Lake Unstructured Data Government 84

Netflix Tech

JANUARY 6, 2025

This process involves: Identifying Stakeholders: Determine who is impacted by the issue and whose input is crucial for a successful resolution. In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. And how did we arrive at thispoint?

Metadata 93

Metadata Algorithm Systems Engineering 93

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction.

Metadata 52

Metadata PostgreSQL Java Database 52

ProjectPro

JUNE 6, 2025

With the global data volume projected to surge from 120 zettabytes in 2023 to 181 zettabytes by 2025, PySpark's popularity is soaring as it is an essential tool for efficient large scale data processing and analyzing vast datasets. Some of the major advantages of using PySpark are- Writing code for parallel processing is effortless.

Hadoop 68

Hadoop Metadata Java Datasets 68

Snowflake

JUNE 10, 2025

Dynamic Tables updates Dynamic Tables provides a declarative processing framework for batch and streaming pipelines. This approach simplifies pipeline configuration, offering automatic orchestration and continuous, incremental data processing. The resulting data can be queried by any Iceberg engine.

Data Pipeline 92

Data Pipeline SQL Python Building 92

Christophe Blefari

MARCH 1, 2023

You can also add metadata on models (in YAML). docs — in dbt you can add metadata on everything, some of the metadata is already expected by the framework and thank to it you can generate a small web page with your light catalog inside: you only need to do dbt docs generate and dbt docs serve.

Data Warehouse 278

Data Warehouse Metadata SQL Raw Data 278

ProjectPro

JUNE 6, 2025

Want to process peta-byte scale data with real-time streaming ingestions rates, build 10 times faster data pipelines with 99.999% reliability, witness 20 x improvement in query performance compared to traditional data lakes, enter the world of Databricks Delta Lake now. This results in a fast and scalable metadata handling system.

Data Lake 63

Data Lake Data Warehouse Metadata BI 63

ProjectPro

JUNE 6, 2025

With an increasing amount of big data, there is a need for a service like ADF that can orchestrate and operationalize processes to refine the enormous stores of raw business data into actionable business insights. Activities: Activities represent a processing step in a pipeline. What are the steps involved in an ETL process?

Data Lake 60

Data Lake Metadata SQL ETL Tools 60

Snowflake

JANUARY 15, 2025

The world of geospatial data processing is vast and complex, and were here to simplify it for you. While you can do time-series forecasting across any time-based data, enriching that forecasting with location data provides another value dimension in the forecasting process. Load the GeoTIFF file. Load the shapefile.

Transportation 97

Transportation BI Database-centric Metadata 97

Cloudera

DECEMBER 19, 2024

A key consideration for customers who find themselves in this scenario is to simplify as much as possible: choose platforms that provide a consistent experience, leverage tools that span multiple environments, and invest in open standards, technologies, and processes to ensure maximum flexibility now and in the future.

AWS 75

AWS Metadata Government Machine Learning 75

Snowflake

MARCH 6, 2025

Customer intelligence teams analyze reviews and forum comments to identify sentiment trends, while support teams process tickets to uncover product issues and inform gaps in a product roadmap. Meanwhile, operations teams use entity extraction on documents to automate workflows and enable metadata-driven analytical filtering.

Unstructured Data 97

Unstructured Data Media Medical Data Workflow 97

Data Engineering Weekly

MARCH 5, 2025

In the mid-2000s, Hadoop emerged as a groundbreaking solution for processing massive datasets. Cost: Reducing storage and processing expenses. Hadoop achieved this through distributed processing and storage, using a framework called MapReduce and the Hadoop Distributed File System (HDFS). Speed: Accelerating data insights.

Hadoop 58

Hadoop Metadata Data Ingestion Data Governance 58

Cloudera

DECEMBER 3, 2024

REST Catalog Value Proposition It provides open, metastore-agnostic APIs for Iceberg metadata operations, dramatically simplifying the Iceberg client and metastore/engine integration. It provides real time metadata access by directly integrating with the Iceberg-compatible metastore. spark.sql(SELECT * FROM airlines_data.carriers).show()

Metadata 84

Metadata SQL Data Warehouse Database 84

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata BI Data Lake Government 84

Engineering at Meta

FEBRUARY 4, 2025

In this context, an individual data log entry is a formatted version of a single row of data from Hive that has been processed to make the underlying data transparent and easy to understand. Once the batch has been queued for processing, we copy the list of user IDs who have made requests in that batch into a new Hive table.

Accessibility 89

Accessibility Accessible Raw Data Data Warehouse 89

Data Engineering Weekly

MAY 25, 2025

Then, a custom Apache Beam consumer processed these events, transforming and writing them to CRDB. link] Vimeo: Behind Viewer Retention Analytics at Scale Vimeo outlines its architecture for delivering viewer retention analytics at scale, leveraging ClickHouse and AI to process data from over a billion videos. and Lite 2.0)

Data Engineering 63

Data Engineering Data Engineer Engineering PostgreSQL 63

Cloudera

OCTOBER 10, 2024

Compaction is a process that rewrites small files into larger ones to improve performance. Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance.

IT 84

IT Data Lake Metadata Data Warehouse 84

Snowflake

DECEMBER 4, 2024

Beyond working with well-structured data in a data warehouse, modern AI systems can use deep learning and natural language processing to work effectively with unstructured and semi-structured data in data lakes and lakehouses.

Data Lake 97

Data Lake Unstructured Data Deep Learning Metadata 97

Precisely

JUNE 10, 2025

Challenge: Manual data quality processes don’t scale for AI and analytics The impact You’re dealing with more data – and complexity – than ever. If you’re still relying on manual processes to match, merge, and resolve data issues, then you’re spending too much time fixing errors and not enough time acting on insights.

Data Integration 52

Data Integration Metadata Data Data Management 52

ProjectPro

JUNE 6, 2025

A machine learning pipeline helps automate machine learning workflows by processing and integrating data sets into a model, which can then be evaluated and delivered. Increased Adaptability and Scope Although you require different models for different purposes, you can use the same functions/processes to build those models.

Machine Learning 58

Machine Learning Building Amazon Web Services Deep Learning 58

Cloudyard

FEBRUARY 17, 2025

To address this, Dynamic CSV Column Mapping with Stored Procedures can be used to create a flexible, automated process that maps additional columns in the CSV to the correct fields in the Snowflake table, making the data loading process smoother and more adaptable. Metadata Proc Step 4: Execute the Stored Procedure.

Metadata 52

Metadata SQL Data Engineering Data Engineer 52

dbt Developer Hub

APRIL 20, 2025

These tools can be called by LLM systems to learn about your data and metadata. For AI agent workflows : Autonomously run dbt processes in response to events. The dbt MCP server provides access to a set of tools that operate on top of your dbt project. Consider starting in a sandbox environment or only granting read permissions.

Structured Data 98

Structured Data SQL BI Metadata 98

ProjectPro

JUNE 6, 2025

Metadata Layer 3. Built to overcome the limitations of other table formats, such as Hive and Parquet , Iceberg offers powerful schema evolution, efficient data processing, ACID compliance, hidden partitioning, and optimized query performance across various compute engines, including Spark, Trino, Flink, and Presto. Iceberg Catalog 2.

Architecture 40

Architecture Data Lake Metadata Cloud Storage 40

ProjectPro

JUNE 6, 2025

An efficient data warehouse schema design can help organizations simplify their decision-making processes, identify growth opportunities, and better understand their business needs or preferences. Plan the ETL process for the data warehouse design. Identify relevant data sources. Define the data destination schema.

Data Warehouse 40

Data Warehouse Designing Metadata Business Intelligence 40

Snowflake

NOVEMBER 12, 2024

Moreover, since no actual data is copied or transferred between accounts — only Snowflake’s services layer and metadata store are used — sharing models reduces the risk of data exposure. Snowflake’s patented cross-cloud technology uses a replication-based approach to enable access to data in remote regions.

AWS 97

AWS Cloud Programming Language High Quality Data 97

ProjectPro

JUNE 6, 2025

It derives its name “Beam” which is from “Batch” + “Stream” from its functionalities for both batch and streaming the parallel processing pipelines for data. It serves as a distributed processing engine for both categories of data streams: unbounded and bounded.

Big Data 49

Big Data Project Metadata Programming Language 49

ProjectPro

JUNE 6, 2025

Conceptual data modeling refers to the process of creating conceptual data models. Physical data modeling is the process of creating physical data models. This is the process of putting a conceptual data model into action and extending it. The process of creating logical data models is known as logical data modeling.

Data Warehouse 53

Data Warehouse NoSQL PostgreSQL Relational Database 53

ProjectPro

JUNE 6, 2025

The two most popular AWS data engineering services for processing data at scale for analytics operations are Amazon EMR and AWS Glue. EMR is a more powerful big data processing solution to provide real-time data streaming for machine learning applications. Executing ETL tasks in the cloud is fast and simple with AWS Glue.

Big Data 52

Big Data AWS Amazon Web Services Project 52