Engineering at Meta

JANUARY 22, 2025

This belief has led us to developing Privacy Aware Infrastructure (PAI) , which offers efficient and reliable first-class privacy constructs embedded in Meta infrastructure to address different privacy requirements, such as purpose limitation , which restricts the purposes for which data can be processed and used. Hack, C++, Python, etc.)

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

ProjectPro

JUNE 6, 2025

” Thus, don't miss out on the opportunity to revolutionize your business with real-time data processing using Azure Stream Analytics. Read this blog till the end to discover how Azure Stream Analytics can simplify your journey to actionable insights. Table of Contents What is Azure Stream Analytics?

Data Process 40

Data Process Process Data Ingestion BI 40

ProjectPro

JUNE 6, 2025

Through this blog, we delve into the fundamental concepts and techniques offered by Pandas, providing you with a comprehensive cheat sheet to navigate the complexities of data preprocessing in Python and Data Science. You will understand how to customize the import process, handle null values, and specify data types during data loading.

Data Process 40

Data Process Process Aggregated Data Data Science 40

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. This process can also be used to track the provenance of increments.

Datasets 103

Datasets Computer Science Kafka Systems 103

Snowflake

NOVEMBER 26, 2024

This blog post is the second in a three-part series on migrations. Processing some 90,000 tables per day, the team oversees the ingestion of more than 100 terabytes of data from upward of 8,500 events daily. That’s why we’ve collected these migration success stories to help you get started on your migration to Snowflake.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Data Lake 105

Snowflake

JANUARY 15, 2025

Over the last three geospatial-centric blog posts, weve covered the basics of what geospatial data is, how it works in the broader world of data and how it specifically works in Snowflake based on our native support for GEOGRAPHY , GEOMETRY and H3. But there is so much more you can do with geospatial data in your Snowflake account!

Transportation 97

Transportation BI Database-centric Metadata 97

Cloudera

NOVEMBER 13, 2024

The end-to-end lineage also automates tasks such as predicting the impact of a process change, analyzing the impact of a broken process, discovering parallel processes performing the same tasks, and performing root cause analysis to uncover the source of reporting errors.

Metadata 90

Metadata Management Data Governance Government 90

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. The evolution of OLAP — What is OLAP in the modern data stack?

Data 130

Data Data Warehouse Programming Language Coding 130

Snowflake

JANUARY 16, 2025

This is done by combining parameter preserving model rewiring with lightweight fine-tuning to minimize the likelihood of knowledge being lost in the process. You can learn more in our SwiftKV research blog post. SwiftKV achieves higher throughput performance with minimal accuracy loss (see Tables 1 and 2).

Algorithm 97

Algorithm Data Analysis Building Process 97

ProjectPro

JUNE 6, 2025

This blog introduces the concept of time series forecasting models in the most detailed form. The blog's last two parts cover various use cases of these models and projects related to time series analysis and forecasting problems. This blog will explore these use cases with practical time series forecasting model examples.

Deep Learning 66

Deep Learning Machine Learning Python Datasets 66

Snowflake

NOVEMBER 11, 2024

Here’s how Snowflake Cortex AI and Snowflake ML are accelerating the delivery of trusted AI solutions for the most critical generative AI applications: Natural language processing (NLP) for data pipelines: Large language models (LLMs) have a transformative potential, but they often batch inference integration into pipelines, which can be cumbersome.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Snowflake

JANUARY 7, 2025

In this blog post, well explain how a composable CDP works and how it can empower financial services marketers to fully leverage their data and AI to improve performance. Composable CDPs support streamlining of governance processes that manage marketer workflows. Use all of your data.

Banking 98

Banking Media Government Cloud 98

ProjectPro

JUNE 6, 2025

This blog offers an Agentic AI learning path, explaining the core components behind AI Agents. If you’ve ever wondered how these intelligent systems work or wanted to build one, this blog is your starting point. Read more about AI agents in our latest blog: AI Agents: The New Human-Like Heroes of AI.

Deep Learning 80

Deep Learning Algorithm Machine Learning Banking 80

ProjectPro

JUNE 6, 2025

Cloud computing is the future, given that the data being produced and processed is increasing exponentially. We will cover all such details in this blog. Assume they also have an order processing service that processes active orders submitted by the customers but not fulfilled by the company. zettabytes in 2020.

AWS 67

AWS Big Data SQL Raw Data 67

DataKitchen

NOVEMBER 27, 2024

This blog delves into the six distinct types of data quality dashboards, examining how each fulfills a specific role in ensuring data excellence. Is completeness about filling every field in a record, or is it about having the fields critical to a particular business process? However, not all data quality dashboards are created equal.

Banking 86

Banking Data Consulting Pharmaceutical 86

Data Engineering Weekly

MAY 25, 2025

The blog is an excellent compilation of types of query engines on top of the lakehouse, its internal architecture, and benchmarking against various categories. Then, a custom Apache Beam consumer processed these events, transforming and writing them to CRDB. link] Gunnar Morling: What If We Could Rebuild Kafka From Scratch?

Data Engineer 62

Data Engineer Data Engineering Engineering PostgreSQL 62

ProjectPro

JUNE 6, 2025

From understanding the delays to implementing effective solutions, dive into practical strategies for optimizing serverless performance in this blog. With the global cloud computing market size likely to reach over $727 billion in 2024 , AWS Lambda has emerged as a game-changer, simplifying complex processes with its serverless architecture.

AWS 58

AWS Amazon Web Services Programming Language Media 58

Monte Carlo

NOVEMBER 26, 2024

Process > Tooling (Barr) 3. Process > Tooling (Barr) A new tool is only as good as the process that supports it. The move toward self-serve AI-enabled pipeline management means that the most painful part of everyone’s job gets automated away—and their ability to create and demonstrate new value expands in the process.

Data Engineer 105

Data Engineer Data Engineering Engineering Unstructured Data 105

Data Engineering Weekly

JUNE 1, 2025

The blog narrates adopting a hybrid approach with AWS Sagemaker integration and Chalk feature store. link] EloElo: Building EloElo’s Data Platform - Our 2-year Journey to Batch + Real-Time Lakehouse on Open Source stack The blog captures the emerging lakehouse architecture, combining CDC pipelines and event sourcing systems.

Data Engineer 70

Data Engineer Data Engineering Engineering Relational Database 70

Snowflake

NOVEMBER 12, 2024

Snowpark Container Services (now generally available on Azure as well as AWS) gives developers the flexibility to easily and securely deploy complex functionality — from custom frontends and large-scale ML training and inference to open source and homegrown models — all securely within Snowflake (learn more in this blog post ).

AWS 97

AWS Cloud Programming Language High Quality Data 97

ProjectPro

JUNE 6, 2025

Data engineering is the foundation for data science and analytics by integrating in-depth knowledge of data technology, reliable data governance and security, and a solid grasp of data processing. This blog covers the top ten AWS data engineering tools popular among data engineers across the big data industry.

AWS 52

AWS Data Engineer Data Engineering Engineering 52

KDnuggets

JUNE 6, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter 5 Error Handling Patterns in Python (Beyond Try-Except) Stop letting errors crash your app.

Python 74

Python Data Science Machine Learning Database 74

ProjectPro

JUNE 6, 2025

Looking for an efficient tool for streamlining and automating your data processing workflows? Read this blog till the end to learn everything you need to know about Airflow DAG. Let's consider an example of a data processing pipeline that involves ingesting data from various sources, cleaning it, and then performing analysis.

Data Pipeline 63

Data Pipeline PostgreSQL Python Database 63

ProjectPro

JUNE 6, 2025

It is extremely important for businesses to process data correctly since the volume and complexity of raw data are rapidly growing. Over the past few years, data-driven enterprises have succeeded with the Extract Transform Load (ETL) process to promote seamless enterprise data exchange.

BI 45

BI ETL Tools Retail Healthcare 45

ProjectPro

JUNE 6, 2025

This transformation is where data warehousing tools come into play, acting as the refining process for your data. These tools are crucial in modern business intelligence and data-driven decision-making processes. Amazon Redshift Pros High query performance via columnar storage and parallel processing.

Data Storage 58

Data Storage PostgreSQL Data Warehouse AWS 58

Netflix Tech

APRIL 8, 2025

By Cheng Xie , Bryan Shultz , and Christine Xu In a previous blog post , we described how Netflix uses eBPF to capture TCP flow logs at scale for enhanced network insights. 2xlarge instances, we can process 5 million flows per second across the entire Netflixfleet. With 30 c7i.2xlarge

AWS 76

AWS Kafka Programming Certification 76

ProjectPro

JUNE 6, 2025

Read this blog if you are interested in exploring business intelligence projects examples that highlight different strategies for increasing business growth. Along with that, deep learning algorithms and image processing methods are also used over medical reports to support a patient’s treatment better. Chilly December is here!

Business Intelligence 69

Business Intelligence Coding Project BI 69

ProjectPro

JUNE 6, 2025

This blog introduces you to AWS DevOps and the various AWS services it offers for cloud computing. If you’re curious to learn why you should leverage these AWS DevOps tools and how different businesses benefit, this blog is for you. Benefits of AWS CodePipeline 1.

AWS 65

AWS Architecture Amazon Web Services Programming Language 65

ProjectPro

JUNE 6, 2025

This blog will help you understand what data engineering is with an exciting data engineering example, why data engineering is becoming the sexier job of the 21st century is, what is data engineering role, and what data engineering skills you need to excel in the industry, Table of Contents What is Data Engineering?

Data Science 40

Data Science Data Engineer Data Engineering Engineering 40

ProjectPro

JUNE 6, 2025

This blog introduces Cache-Augmented Generation (CAG), a powerful alternative to RAG for enhancing LLM responses. These documents are processed and stored in a Key-Value (KV) Cache, which acts like a fast-access memory bank for the model. You can find in-depth blogs on these topics on the ProjectPro website.

Manufacturing 52

Manufacturing Database Datasets Systems 52

ProjectPro

JUNE 6, 2025

Did you know Apache Kafka was the leading technology in the global big data processing business in 2023, with a 16.88 This blog is the ideal roadmap for big data professionals looking for the top resources to master Apache Kafka. These events are crucial in facilitating real-time streaming and data processing.

Kafka 40

Kafka Java Big Data Data Pipeline 40

Pinterest Engineering

APRIL 7, 2025

In this blog post, well discuss our experiences in identifying the challenges associated with EC2 network throttling. In the remainder of this blog post, well share how we root cause and mitigate the aboveissues. This prompted us to engage with AWS and dive deep into the network performance of our clusters. 4xl with up to 12.5

AWS 63

AWS Bytes Data Ingestion Database 63

ProjectPro

JUNE 6, 2025

Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. billion by 2026?

AWS 66

AWS Scala Metadata Data Lake 66

Engineering at Meta

APRIL 28, 2025

Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process. However, conducting these processes outside of developer workflows presented challenges in terms of accuracy and timeliness.

Metadata 83

Metadata Data Utilities Data Warehouse 83

ProjectPro

JUNE 6, 2025

ETL is considered to be an essential part of data warehousing architecture in business processes worldwide. Although every interview is unique and every role has a different range of responsibilities, this blog can help you with the top ten ETL Interview Questions that will enable you to take the leap and excel in your interview.

Data Engineer 45

Data Engineer Data Engineering ETL Tools Engineering 45

ProjectPro

JUNE 6, 2025

With the help of natural language processing (NLP) tools, it has led to the development of exciting artificial intelligence applications like language recognition, autonomous vehicles, and computer vision robots, to name a few. Deep Learning also uses the same analogy of a brain neuron for processing the information and recognizing them.

Deep Learning 73

Deep Learning Algorithm Machine Learning Datasets 73

ProjectPro

JUNE 6, 2025

Check out this blog post that lists the seven best PyTorch books that are an essential read for data science beginners like Steve and experienced developers. But with so much to learn and explore in PyTorch, it is difficult to know where to start.

Deep Learning 53

Deep Learning Machine Learning Data Science Coding 53