Engineering at Meta

FEBRUARY 4, 2025

Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand.

Accessible 81

Accessible Accessibility Raw Data Data Warehouse 81

Analytics Vidhya

JUNE 12, 2023

Introduction Meet Tajinder, a seasoned Senior Data Scientist and ML Engineer who has excelled in the rapidly evolving field of data science. Tajinder’s passion for unraveling hidden patterns in complex datasets has driven impactful outcomes, transforming raw data into actionable intelligence.

Machine Learning 185

Machine Learning Engineering Raw Data Data Science 185

KDnuggets

JUNE 27, 2022

If your raw data is in a SQL-based data lake, why spend the time and money to export the data into a new platform for data prep?

Data Preparation 160

Data Preparation SQL Raw Data Data Lake 160

Analytics Vidhya

FEBRUARY 25, 2023

Introduction A data lake is a centralized and scalable repository storing structured and unstructured data. The need for a data lake arises from the growing volume, variety, and velocity of data companies need to manage and analyze.

Cloud Storage 162

Cloud Storage Data Lake Cloud Unstructured Data 162

Ascend.io

OCTOBER 28, 2024

What is Data Transformation? Data transformation is the process of converting raw data into a usable format to generate insights. It involves cleaning, normalizing, validating, and enriching data, ensuring that it is consistent and ready for analysis.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision. Enter the ELT.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Hevo

DECEMBER 6, 2024

Data preparation tools are very important in the analytics process. They transform raw data into a clean and structured format ready for analysis. These tools simplify complex data-wrangling tasks like cleaning, merging, and formatting, thus saving precious time for analysts and data teams. 

Data Preparation 52

Data Preparation Raw Data Data Process 52

DataKitchen

DECEMBER 6, 2024

Would you like help maintaining high-quality data across every layer of your Medallion Architecture? Like an Olympic athlete training for the gold, your data needs a continuous, iterative process to maintain peak performance. Want More Detail? Read the popular blog article.

Architecture 52

Architecture Raw Data High Quality Data Data Validation 52

Snowflake

JANUARY 29, 2024

Imagine seamlessly integrating your Google Analytics data with Snowflake, allowing you to combine it effortlessly with other key sources like CRM, ERP, social media metrics, email campaign data, and whatever data sources compose the full scope of your data estate. Here’s a quick guide to get started: 1.

Raw Data 126

Raw Data Aggregated Data Cloud Data 126

The Pragmatic Engineer

OCTOBER 17, 2024

Storing data: data collected is stored to allow for historical comparisons. Benchmarking: for new server types identified – or ones that need an updated benchmark executed to avoid data becoming stale – those instances have a benchmark started on them. Each benchmarking task is evaluated sequentially.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Cloudyard

JANUARY 15, 2025

Read Time: 2 Minute, 33 Second Snowflakes PARSE_DOCUMENT function revolutionizes how unstructured data, such as PDF files, is processed within the Snowflake ecosystem. However, Ive taken this a step further, leveraging Snowpark to extend its capabilities and build a complete data extraction process. Why Use PARSE_DOC?

Data Cleanse 52

Data Cleanse Insurance Raw Data Unstructured Data 52

Cloudyard

DECEMBER 24, 2024

Read Time: 2 Minute, 11 Second In today’s data-driven world, organizations demand powerful tools to transform, analyze, and present their data seamlessly. They need to: Consolidate raw data from orders, customers, and products. Enrich and clean data for downstream analytics. Develop a VIEW in Semantic Layer.

Building 52

Building Raw Data Scala Business Intelligence 52

databricks

AUGUST 2, 2024

Training a high-quality machine learning model requires careful data and feature preparation. To fully utilize raw data stored as tables in Databricks, running.

Engineering 135

Engineering Raw Data Machine Learning Utilities 135

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. The data warehouse needs to reflect the business, and the business should have clarity on how it thinks about analytics.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineering 174

KDnuggets

JULY 18, 2023

Data Storytelling with Python Altair and Generative AI teaches you how to turn raw data into effective, insightful data stories. You’ll learn exactly what goes into an effective data story, then combine your Python data skills with the Altair library and AI tools to rapidly create amazing visualizations.

Raw Data 105

Raw Data Python Data 105

Precisely

NOVEMBER 20, 2023

The answers lie in data integrity and the contextual richness of the data that fuels your AI. Businesses must navigate many legal and regulatory requirements, including data privacy laws, industry standards, security protocols, and data sovereignty requirements. Contextual data. Data integrity is multifaceted.

Raw Data 121

Raw Data Insurance Data Portfolio 121

Maxime Beauchemin

JANUARY 7, 2018

Batch data processing  — historically known as ETL —  is extremely challenging. In this post, we’ll explore how applying the functional programming paradigm to data engineering can bring a lot of clarity to the process. It’s time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot.

Data Process 183

Data Process Data Engineering Data Engineer Process 183

Cloudera

JANUARY 9, 2025

A Name That Matches the Moment For years, Clouderas platform has helped the worlds most innovative organizations turn data into action. But over the years, data teams and data scientists overcame these hurdles and AI became an engine of real-world innovation. Thats why were moving from Cloudera Machine Learning to Cloudera AI.

Machine Learning 85

Machine Learning Raw Data Government Algorithm 85

Towards Data Science

FEBRUARY 9, 2023

This post is for those poor souls that need to scan terabytes of data in BigQuery to calculate some counts, sums, or rolling totals over huge event data on a daily or even at a higher frequency basis. In this post, I will go over a technique for enabling a cheap data injestion and cheap data consumption for “big data”.

Big Data 98

Big Data Raw Data Aggregated Data Data 98

InData Labs

OCTOBER 17, 2023

Big data is central to the efficient running of all modern organizations, but to be of use, raw data must be suitably organized. Запись The benefits of modern data architecture впервые появилась InData Labs. Запись The benefits of modern data architecture впервые появилась InData Labs.

Data Architecture 105

Data Architecture Architecture Raw Data Big Data 105

Monte Carlo

NOVEMBER 21, 2024

Let’s set the scene: your company collects data, and you need to do something useful with it. Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Precisely

SEPTEMBER 25, 2023

77% of data and analytics professionals say data-driven decision-making is the top goal for their data programs. Data-driven decision-making and initiatives are certainly in demand, but their success hinges on … well, the data that supports them. More specifically, the quality and integrity of that data.

Data Validation 100

Data Validation Process Raw Data Data Cleanse 100

Towards Data Science

NOVEMBER 16, 2023

Data Management A tutorial on how to use VDK to perform batch data processing Photo by Mika Baumeister on Unsplash Versatile Data Ki t (VDK) is an open-source data ingestion and processing framework designed to simplify data management complexities.

Data Process 98

Data Process Process Raw Data Data 98

Towards Data Science

FEBRUARY 6, 2024

This generalisation makes their data models complex and cryptic and require domain expertise. Even harder to manage, a common setup within large organisations is to have several instances of these systems with some underlaying processes in charge of transmitting data among them, which could lead to duplications, inconsistencies, and opacity.

Systems 98

Systems Raw Data Metadata Data Cleanse 98

Snowflake

NOVEMBER 11, 2024

At Snowflake BUILD , we are introducing powerful new features designed to accelerate building and deploying generative AI applications on enterprise data, while helping you ensure trust and safety. These scalable models can handle millions of records, enabling you to efficiently build high-performing NLP data pipelines.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

KDnuggets

NOVEMBER 17, 2021

With the customer at its heart, modern augmented BI platforms no longer require scripting/coding skills or the knowledge to build the back-end data models, empowering even laymen to harness the power of raw data. As a user, here are the top AI capabilities that you need to look for in BI software.

BI 134

BI Raw Data Coding Skills Coding 134

Christophe Blefari

APRIL 21, 2023

A lot of data teams embraced dbt, or at least the SQL with engineering practices to transform data in cloud data warehouses. It is interesting to read this post jointly with the future of data engineer at Meta. Data Economy 💰 Betterdata raises $1.65m seed round. Synthetic data are AI generated data.

Raw Data 130

Raw Data Data SQL Datasets 130

The Pragmatic Engineer

JANUARY 12, 2023

We cover one and a half out of eight topics in today’s subscriber-only issue, Inside Pollen's Transparent Compensation Data. In this week’s The Scoop, I analyzed this information and dissected it, going well beyond the raw data. How can you use this data in budgeting, and what are the caveats to be aware of?

Software Engineering 219

Software Engineering Software Engineer Engineering Raw Data 219

Hevo

FEBRUARY 23, 2025

Organizations generate tons of data every second, yet 80% of enterprise data remains unstructured and unleveraged (Unstructured Data). Organizations need data ingestion and integration to realize the complete value of their data assets.

Data Ingestion 40

Data Ingestion Data Integration Unstructured Data Raw Data 40

Hevo

DECEMBER 6, 2024

Data integration is an integral part of modern business strategy, enabling businesses to convert raw data into actionable information and make data-driven decisions. However, its technical complexities and steeper learning curve can create a challenge for teams that require an efficient real-time data pipeline.

Raw Data 52

Raw Data Data Pipeline Data Integration Data 52

RandomTrees

MARCH 10, 2025

Using Data to Gain Future Knowledge In order to evaluate past data and forecast future events, predictive analytics makes use of statistical models, machine learning, and data mining. Spotify offers hyper-personalized experiences for listeners by analysing user data.

Retail 52

Retail Hospitality Data Governance Banking 52

Hevo

FEBRUARY 23, 2025

Organizations generate tons of data every second, yet 80% of enterprise data remains unstructured and unleveraged (Unstructured Data). Organizations need data ingestion and integration to realize the complete value of their data assets.

Data Ingestion 40

Data Ingestion Data Integration Unstructured Data Raw Data 40

Engineering at Meta

JANUARY 21, 2025

When you combine talented engineers with rich performance data you can get efficiency wins by both creating tooling to identify issues before they reach production and finding opportunities in already running code. Metas existing tools can identify the issue and query Strobelight data to estimate the impact on compute cost.

Technology 115

Technology Metadata Utilities Engineering 115

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 105

Engineering Raw Data Data Science Machine Learning 105

DataKitchen

AUGUST 9, 2021

Below is our third post (3 of 5) on combining data mesh with DataOps to foster greater innovation while addressing the challenges of a decentralized architecture. We’ve talked about data mesh in organizational terms (see our first post, “ What is a Data Mesh? ”) and how team structure supports agility. Source: Thoughtworks.

Pharmaceutical 98

Pharmaceutical Raw Data Data Data Lake 98