Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 120

Data Warehouse SQL Programming Language Data 120

Precisely

MARCH 3, 2025

Data quality and governance are critical. Without clean, governed data, automation efforts can be undermined, impacting your business outcomes and AI initiatives. Data and process automation used to be seen as luxury but those days are gone.

Process 58

Process Government Data Finance 58

Christophe Blefari

JANUARY 11, 2025

The Data News are here to stay, the format might vary during the year, but here we are for another year. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. HNY 2025 ( credits ) Happy new year ✨ I wish you the best for 2025. Not really digest.

Data 130

Data Data Warehouse Coding Programming Language 130

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. It’s the big blueprint we data engineers follow in order to transform raw data into valuable insights.

Architecture 96

Architecture Data Engineer Data Engineering Engineering 96

Monte Carlo

NOVEMBER 26, 2024

Here’s where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024—plus a few predictions of my own. 2025 data engineering trends incoming. Process > Tooling (Barr) 3. Small data is the future of AI (Tomasz) 7. The lines are blurring for analysts and data engineers (Barr) 8.

Data Engineer 105

Data Engineer Data Engineering Engineering Unstructured Data 105

Analytics Vidhya

FEBRUARY 28, 2023

Introduction Data science has taken over all economic sectors in recent times. To achieve maximum efficiency, every company strives to use various data at every stage of its operations.

Data Science 290

Data Science Data Preparation Big Data Data 290

Striim

MARCH 21, 2025

What is Real-Time Stream Processing? In today’s fast-moving world, companies need to glean insights from data as soon as it’s generated. To access real-time data, organizations are turning to stream processing. To access real-time data, organizations are turning to stream processing.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Precisely

JANUARY 30, 2025

Key Takeaways: Prioritize metadata maturity as the foundation for scalable, impactful data governance. Recognize that artificial intelligence is a data governance accelerator and a process that must be governed to monitor ethical considerations and risk.

Data Governance 102

Data Governance Government Metadata Data 102

Waitingforcode

FEBRUARY 18, 2025

From another, they often have quotas and limits that you, as a data engineer, have to take into account in your daily work. These limits become even more serious when they operate in a latency-sensitive context, as the one of stream processing.

Data Engineering 130

Data Engineering Data Engineer Cloud Data 130

Cloudera

NOVEMBER 13, 2024

We are excited to announce the acquisition of Octopai , a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. This dampens confidence in the data and hampers access, in turn impacting the speed to launch new AI and analytic projects.

Metadata 90

Metadata Management Data Governance Government 90

Engineering at Meta

APRIL 28, 2025

Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI).

Metadata 86

Metadata Data Utilities Data Warehouse 86

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineer 248

Data Engineer Data Engineering Engineering Data 248

Towards Data Science

DECEMBER 10, 2024

Three Zero-Cost Solutions That Take Hours, NotMonths A data quality certified pipeline. Source: unsplash.com In my career, data quality initiatives have usually meant big changes. From governance processes to costly tools to dbt implementationdata quality projects never seem to want to besmall. Created by the author using draw.io

PostgreSQL 75

PostgreSQL Data Python SQL 75

Start Data Engineering

SEPTEMBER 18, 2024

Parts of data engineering 3.1. Identify what tool to use to process data 3.3. Data flow architecture 3. Introduction 2. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4.

Project 240

Project Building Datasets Architecture 240

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 68

Data Architecture Architecture Metadata Government 68

Engineering at Meta

FEBRUARY 4, 2025

Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand.

Accessible 92

Accessible Accessibility Raw Data Data Warehouse 92

Christophe Blefari

MARCH 15, 2024

Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. Pandera, a data validation library for dataframes, now supports Polars.

Metadata 272

Metadata Data Software Engineer Software Engineering 272

DataKitchen

NOVEMBER 5, 2024

The Race For Data Quality In A Medallion Architecture The Medallion architecture pattern is gaining traction among data teams. It is a layered approach to managing and transforming data. By systematically moving data through these layers, the Medallion architecture enhances the data structure in a data lakehouse environment.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Analytics Vidhya

FEBRUARY 23, 2023

Introduction Big Data is a large and complex dataset generated by various sources and grows exponentially. It is so extensive and diverse that traditional data processing methods cannot handle it. The volume, velocity, and variety of Big Data can make it difficult to process and analyze.

Big Data Tools 242

Big Data Tools Big Data Datasets Data 242

Snowflake

NOVEMBER 26, 2024

Many of our customers — from Marriott to AT&T — start their journey with the Snowflake AI Data Cloud by migrating their data warehousing workloads to the platform. Today we’re focusing on customers who migrated from a cloud data warehouse to Snowflake and some of the benefits they saw. million in cost savings annually.

Data Warehouse 101

Data Warehouse Cloud PostgreSQL Hadoop 101

Start Data Engineering

MAY 28, 2024

Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Introduction 2. Project demo 3. Use DuckDB 4.4.

Data Pipeline 219

Data Pipeline Python Building Data 219

Snowflake

MARCH 26, 2025

In this edition, we talk to Richard Meng, co-founder and CEO of ROE AI , a startup that empowers data teams to extract insights from unstructured, multimodal data including documents, images and web pages using familiar SQL queries. I experienced the thrilling pace of AI data innovation firsthand.

Unstructured Data 95

Unstructured Data SQL Data Data Workflow 95

Towards Data Science

DECEMBER 16, 2024

Agentic AI, small data, and the search for value in the age of the unstructured datastack. Heres where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024plus a few predictions of myown. 2025 data engineering trends incoming. Search: tools that leverage a corpus of data to answer questions 3.

Unstructured Data 80

Unstructured Data Data Food Data Engineering 80

Analytics Vidhya

AUGUST 7, 2024

Introduction Apache Airflow is a crucial component in data orchestration and is known for its capability to handle intricate workflows and automate data pipelines. Many organizations have chosen it due to its flexibility and strong scheduling capabilities.

Data Pipeline 192

Data Pipeline Data Process Data Workflow 192

Analytics Vidhya

FEBRUARY 28, 2024

Introduction Data is fuel for the IT industry and the Data Science Project in today’s online world. IT industries rely heavily on real-time insights derived from streaming data sources. Handling and processing the streaming data is the hardest work for Data Analysis.

MongoDB 222

MongoDB Data Pipeline Kafka Building 222

Precisely

APRIL 7, 2025

We often use different terms when were talking about the same thing in this case, data appending vs. data enrichment. Ive noticed that “data appending” is more commonly used in industries like marketing and telecommunications, while data enrichment seems to be the preferred term in financial services and retail.

Retail 74

Retail Datasets Data Portfolio 74

dbt Developer Hub

APRIL 20, 2025

dbt is the standard for creating governed, trustworthy datasets on top of your structured data. We expect that over the coming years, structured data is going to become heavily integrated into AI workflows and that dbt will play a key role in building and provisioning this data. What is MCP? Why does this matter?

Structured Data 98

Structured Data SQL BI Project 98

Data Engineering Weekly

MAY 25, 2025

Dagster Components is now here Components provides a modular architecture that enables data practitioners to self-serve while maintaining engineering quality. Understanding this fact will help data tools break new ground with the advancement of AI agents. and Lite 2.0) to pinpoint drop-offs and high retention sections.

Data Engineer 62

Data Engineer Data Engineering Engineering PostgreSQL 62

Snowflake

APRIL 13, 2025

Managing and utilizing data effectively is crucial for organizational success in today's fast-paced technological landscape. The vast amounts of data generated daily require advanced tools for efficient management and analysis. Enter agentic AI, a type of artificial intelligence set to transform enterprise data management.

Data Management 83

Data Management Management Consulting Unstructured Data 83

Snowflake

JANUARY 30, 2025

Use cases range from getting immediate insights from unstructured data such as images, documents and videos, to automating routine tasks so you can focus on higher-value work. Gen AI makes this all easy and accessible because anyone in an enterprise can simply interact with data by using natural language.

Unstructured Data 91

Unstructured Data Entertainment Healthcare Telecommunication 91