Engineering at Meta

JANUARY 22, 2025

This belief has led us to developing Privacy Aware Infrastructure (PAI) , which offers efficient and reliable first-class privacy constructs embedded in Meta infrastructure to address different privacy requirements, such as purpose limitation , which restricts the purposes for which data can be processed and used. Hack, C++, Python, etc.)

Data Warehouse 118

Data Warehouse SQL Programming Language Data 118

Cloudera

AUGUST 6, 2024

And for that future to be a reality, data teams must shift their attention to metadata, the new turf war for data. The need for unified metadata While open and distributed architectures offer many benefits, they come with their own set of challenges. Data teams actually need to unify the metadata. Open data is the future.

Metadata 82

Metadata Government Datasets Architecture 82

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. Arroyo, a stream-processing platform, rebuilt their engine using DataFusion. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.

Metadata 272

Metadata Data Datasets Data Warehouse 272

Engineering at Meta

JANUARY 21, 2025

Strobelight is also not a single profiler but an orchestrator of many different profilers (even ad-hoc ones) that runs on all production hosts at Meta, collecting detailed information about CPU usage, memory allocations, and other performance metrics from running processes. Did someone say Metadata?

Technology 117

Technology Metadata Utilities Engineering 117

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 96

Data Process Process Metadata Finance 96

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction.

Metadata 52

Metadata PostgreSQL Java Database 52

Netflix Tech

JANUARY 10, 2024

Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. The Netflix video processing pipeline went live with the launch of our streaming service in 2007. The Netflix video processing pipeline went live with the launch of our streaming service in 2007.

Process 98

Process Pipeline-centric Media Metadata 98

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The CSP engine is powered by Apache Flink, which is the best-in-class processing engine for stateful streaming pipelines. Currently, Iceberg support in CSP is in technical preview mode.

Process 114

Process SQL Kafka Database 114

Netflix Tech

JANUARY 6, 2025

This process involves: Identifying Stakeholders: Determine who is impacted by the issue and whose input is crucial for a successful resolution. In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. And how did we arrive at thispoint?

Metadata 91

Metadata Algorithm Systems Building 91

Christophe Blefari

MARCH 1, 2023

You can also add metadata on models (in YAML). docs — in dbt you can add metadata on everything, some of the metadata is already expected by the framework and thank to it you can generate a small web page with your light catalog inside: you only need to do dbt docs generate and dbt docs serve.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Cloudera

APRIL 2, 2024

I am pleased to announce that Cloudera has achieved FedRAMP “In Process”, a significant milestone that underscores our commitment to providing the public sector with secure and reliable data management solutions across on-prem, hybrid and multi-cloud environments.

Designing 101

Designing Process Government Metadata 101

Data Engineering Weekly

MARCH 5, 2025

In the mid-2000s, Hadoop emerged as a groundbreaking solution for processing massive datasets. Cost: Reducing storage and processing expenses. Hadoop achieved this through distributed processing and storage, using a framework called MapReduce and the Hadoop Distributed File System (HDFS). Speed: Accelerating data insights.

Hadoop 57

Hadoop Metadata Data Ingestion Data Governance 57

Data Engineering Podcast

NOVEMBER 27, 2022

This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Stale dashboards?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Data Engineering Weekly

MAY 25, 2025

Then, a custom Apache Beam consumer processed these events, transforming and writing them to CRDB. link] Vimeo: Behind Viewer Retention Analytics at Scale Vimeo outlines its architecture for delivering viewer retention analytics at scale, leveraging ClickHouse and AI to process data from over a billion videos. and Lite 2.0)

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Engineering at Meta

FEBRUARY 4, 2025

In this context, an individual data log entry is a formatted version of a single row of data from Hive that has been processed to make the underlying data transparent and easy to understand. Once the batch has been queued for processing, we copy the list of user IDs who have made requests in that batch into a new Hive table.

Accessible 91

Accessible Accessibility Raw Data Data Warehouse 91

LinkedIn Engineering

JUNE 15, 2023

To remove this bottleneck, we built AvroTensorDataset , a TensorFlow dataset for reading, parsing, and processing Avro data. If greater than one, records in files are processed in parallel. The bytes are decoded based on the provided features metadata (i.e. Zero shuffle buffer size means shuffle is disabled.

Datasets 102

Datasets Bytes Process Data Ingestion 102

Snowflake

MARCH 6, 2025

Customer intelligence teams analyze reviews and forum comments to identify sentiment trends, while support teams process tickets to uncover product issues and inform gaps in a product roadmap. Meanwhile, operations teams use entity extraction on documents to automate workflows and enable metadata-driven analytical filtering.

Unstructured Data 90

Unstructured Data Medical Media Data Workflow 90

Cloudyard

FEBRUARY 17, 2025

To address this, Dynamic CSV Column Mapping with Stored Procedures can be used to create a flexible, automated process that maps additional columns in the CSV to the correct fields in the Snowflake table, making the data loading process smoother and more adaptable. Metadata Proc Step 4: Execute the Stored Procedure.

Metadata 52

Metadata SQL Data Engineering Data Engineer 52

Netflix Tech

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Process 91

Process Data Pipeline Datasets SQL 91

Snowflake

DECEMBER 4, 2024

Beyond working with well-structured data in a data warehouse, modern AI systems can use deep learning and natural language processing to work effectively with unstructured and semi-structured data in data lakes and lakehouses.

Unstructured Data 91

Unstructured Data Data Lake Deep Learning Structured Data 91

dbt Developer Hub

APRIL 20, 2025

These tools can be called by LLM systems to learn about your data and metadata. For AI agent workflows : Autonomously run dbt processes in response to events. The dbt MCP server provides access to a set of tools that operate on top of your dbt project. Consider starting in a sandbox environment or only granting read permissions.

Structured Data 98

Structured Data SQL BI Project 98

LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Cloudera

NOVEMBER 13, 2020

Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? Metadata Caching. In the previous design each Impala coordinator daemon kept an entire copy of the contents of the catalog cache in memory and had to be explicitly notified of any external metadata changes.

Metadata 143

Metadata Coding SQL Database 143

LinkedIn Engineering

MARCH 23, 2023

Co-Authors: Yuhong Cheng , Shangjin Zhang , Xinyu Liu, and Yi Pan Efficient data processing is crucial in reducing learning curves, simplifying maintenance efforts, and decreasing operational complexity. By unifying these pipelines, we have saved 94% of processing time. Samza , Spark and Apache Flink ).

Process 97

Process Lambda Architecture Kafka Datasets 97

phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. These systems are built on open standards and offer immense analytical and transactional processing flexibility. Why should we use it? Why are They Essential?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Edureka

MAY 27, 2025

The architecture of Microsoft Fabric is based on several essential elements that work together to simplify data processes: 1. Synapse Data Warehouse Fabric’s enterprise-class data warehouse facilitates deep integration with OneLake, distributed processing, and massive parallelism.

Architecture 52

Architecture BI Business Intelligence Raw Data 52

Ascend.io

DECEMBER 4, 2024

Examples include “reduce data processing time by 30%” or “minimize manual data entry errors by 50%.” Start Small and Scale: Instead of overhauling all processes at once, identify a small, manageable project to automate as a proof of concept. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Start Data Engineering

NOVEMBER 21, 2024

Understand how the platforms process data 3.1.1. Metadata catalog stores information about datasets 3.1.3. Analytical databases aggregate large amounts of data 3. Most platforms enable you to do the same thing but have different strengths 3.1. A compute engine is a system that transforms data 3.1.2.

Metadata 130

Metadata Datasets SQL Database 130

Precisely

FEBRUARY 26, 2025

These enhancements improve data accessibility, enable business-friendly governance, and automate manual processes. Scalable AI/ML functionality – Efficiently scale AI usage within the Suite by leveraging external LLMs, with processing handled by the infrastructure where the model resides.

Data Integration 62

Data Integration Data Management Management Data Governance 62

Netflix Tech

DECEMBER 3, 2022

Obviously not all tools are made with the same use case in mind, so we are planning to add more code samples for other (than classical batch ETL) data processing purposes, e.g. Machine Learning model building and scoring. Attributes are set via Metacat , which is a Netflix internal metadata management platform. test_sparksql_write.py

Data Pipeline 145

Data Pipeline Scala Metadata Food 145

phData: Data Engineering

NOVEMBER 1, 2024

VP of Architecture, Healthcare Industry Organizations will focus more on metadata tagging of existing and new content in the coming years. Get the Trendbook Data Governance Maturity For many organizations, data governance is still in the early stages, focusing on defining policies and processes. No problem! is fairly advanced.

Government 52

Government Data Governance Finance Metadata 52

Data Engineering Podcast

DECEMBER 18, 2022

Summary One of the reasons that data work is so challenging is because no single person or team owns the entire process. This introduces friction in the process of collecting, processing, and using data. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 110

Data Engineering Data Engineer Engineering Metadata 110

Data Engineering Weekly

FEBRUARY 18, 2025

Fluss is a compelling new project in the realm of real-time data processing. It works with streaming processing like Flink and Lakehouse formats like Iceberg and Paimon. Fluss focuses on storing streaming data and does not offer streaming processing capabilities. It excels in event-driven architectures and data pipelines.

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Jesse Anderson

NOVEMBER 14, 2023

That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.

Metadata 147

Metadata Data Science Big Data Data 147

Snowflake

JANUARY 23, 2024

Behind the scenes, Snowpark ML parallelizes data processing operations by taking advantage of Snowflake’s scalable computing platform. For Snowpark ML Operations, the Snowpark Model Registry allows customers to securely manage and execute models in Snowflake, regardless of origin.

Machine Learning 136

Machine Learning Metadata Python Telecommunication 136