Engineering at Meta

JANUARY 22, 2025

In order to build high-quality data lineage, we developed different techniques to collect data flow signals across different technology stacks: static code analysis for different languages, runtime instrumentation, and input and output data matching, etc. web endpoints, data tables, AI models) used across Meta.

Data Warehouse 118

Data Warehouse SQL Programming Language Data 118

Knowledge Hut

JUNE 26, 2023

If you want to break into the field of data engineering but don't yet have any expertise in the field, compiling a portfolio of data engineering projects may help. Data pipeline best practices should be shown in these initiatives. Source Code: Stock and Twitter Data Extraction Using Python, Kafka, and Spark 2.

Data Engineer 98

Data Engineer Data Engineering Coding Project 98

KDnuggets

NOVEMBER 12, 2019

While Pandas is the library for data processing in Python, it isn't really built for speed. Learn more about the new library, Modin, developed to distribute Pandas' computation to speedup your data prep.

Coding 123

Coding Python Data Process Process 123

Snowflake

JUNE 5, 2024

Snowflake AI & ML Studio for LLMs (private preview): Enable users of all technical levels to utilize AI with no-code development. Using Snowflake data processing infrastructure, the service is kept up-to-date with the latest information by automating continuous refreshes as new documents are generated.

Coding 112

Coding Building Management Government 112

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests. Kent Beck ?

Coding 97

Coding SQL Software Engineer Software Engineering 97

Data Engineering Podcast

FEBRUARY 20, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Go to dataengineeringpodcast.com/oxylabs today and use code DEP25 to get your special discount on residential proxies.

Python 100

Python Data Process IT Building 100

Analytics Vidhya

APRIL 28, 2023

It is a famous Scala-coded data processing tool that offers low latency, extensive throughput, and a unified platform to handle the data in real-time. Introduction Apache Kafka is an open-source publish-subscribe messaging application initially developed by LinkedIn in early 2011.

Kafka 206

Kafka Scala Coding Data Process 206

The Pragmatic Engineer

MAY 1, 2024

The company says: “Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. years ago, and it became the leading AI coding assistant almost overnight. It’s more a copilot.

Software Engineer 239

Software Engineer Software Engineering Programming Language Media 239

Confluent

MARCH 18, 2025

Confluent for VS Code streamlines workflows, accelerates development cycles, and enhances real-time data processing, all within a unified environment.

Coding 72

Coding Project Data Data Process 72

Snowflake

JUNE 5, 2024

Snowflake customers are already harnessing the power of Python through Snowpark , a set of libraries and code execution environments that run Python and other programming languages next to your data in Snowflake. pandas is the go-to data processing library for millions worldwide, including countless Snowflake users.

Python 128

Python Programming Language Government SQL 128

Snowflake

NOVEMBER 11, 2024

These scalable models can handle millions of records, enabling you to efficiently build high-performing NLP data pipelines. However, scaling LLM data processing to millions of records can pose data transfer and orchestration challenges, easily addressed by the user-friendly SQL functions in Snowflake Cortex.

Unstructured Data 97

Unstructured Data SQL AWS Healthcare 97

Data Engineering Podcast

NOVEMBER 27, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Process 100

Data Process Process Metadata Business Intelligence 100

Netflix Tech

DECEMBER 17, 2024

Metric definitions are often scattered across various databases, documentation sites, and code repositories, making it difficult for analysts and data scientists to find reliable information quickly. Besides providing the end user with an instant answer in a preferred data visualization, LORE instantly learns from the users feedback.

Engineering 85

Engineering Entertainment Amazon Web Services Utilities 85

Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. Search for "Code Commentst" in your podcast player or go to dataengineeringpodcast.com/codecomments today to subscribe.

Process 147

Process Data Lake High Quality Data Machine Learning 147

Snowflake

JUNE 20, 2024

Designed for processing large data sets, Spark has been a popular solution, yet it is one that can be challenging to manage, especially for users who are new to big data processing or distributed systems. It provided us insights as to code compatibility and allowed us to better estimate our migration time.”

Data Engineering 110

Data Engineering Data Engineer Scala Engineering 110

Start Data Engineering

SEPTEMBER 4, 2024

Key parts of data systems: 2.1. Data flow design 2.3. Data processing design 2.5. Code organization 2.6. Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Introduction 2.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Storage 130

Start Data Engineering

OCTOBER 14, 2024

Code & Data 3. Using nested data types effectively 3.1. Using nested data types in data processing 3.3.1. STRUCT enables more straightforward data schema and data access 3.3.2. Nested data types can be sorted 3.3.3. Introduction 2.

SQL 130

SQL Data Schemas Data Coding 130

DataKitchen

NOVEMBER 5, 2024

The Race For Data Quality In A Medallion Architecture The Medallion architecture pattern is gaining traction among data teams. It is a layered approach to managing and transforming data. By systematically moving data through these layers, the Medallion architecture enhances the data structure in a data lakehouse environment.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Snowflake

FEBRUARY 13, 2025

Snowflake has embraced serverless since our founding in 2012, with customers providing their code to load, manage and query data and us taking care of the rest. They can easily access multiple code interfaces, including those for SQL and Python, and the Snowflake AI & ML Studio for no-code development.

Management 84

Management Government Cloud Unstructured Data 84

Snowflake

JUNE 4, 2024

Snowflake’s new Python API (GA soon) simplifies data pipelines and is readily available through pip install snowflake. Automate or code, the choice is yours. Finally, Tasks Backfill (PrPr) automates historical data processing within Task Graphs. Interact with Snowflake objects directly in Python.

Data Pipeline 121

Data Pipeline Python SQL Database 121

Netflix Tech

DECEMBER 3, 2022

One of the main reasons this feature exists is just like with food samples, to give you “a taste” of the production quality ETL code that you could encounter inside the Netflix data ecosystem. " , country_code STRING COMMENT "Country code of the playback session." Let’s review the transformation steps below.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Waitingforcode

FEBRUARY 16, 2023

I like writing code and each time there is a data processing job to write with some business logic I'm very happy. However, with time I've learned to appreciate the Open Source contributions enhancing my daily work. Mack library, the topic of this blog post, is one of those projects discovered recently.

Coding 130

Coding Data Process Project Process 130

RandomTrees

APRIL 18, 2024

Among the various tools available for data integration, Informatica and Talend stand out as popular choices, each with its strengths and capabilities. However, migrating from one platform to another can be a daunting task, especially when it involves converting existing code. Customizable: Tailors to specific project needs and rules.

Coding 52

Coding Retail Metadata Python 52

Ascend.io

DECEMBER 4, 2024

Examples include “reduce data processing time by 30%” or “minimize manual data entry errors by 50%.” It aims to streamline and automate data workflows, enhance collaboration and improve the agility of data teams. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Weekly

APRIL 20, 2025

[link] Jing Ge: Context Matters — The Vision of Data Analytics and Data Science Leveraging MCP and A2A All aspects of software engineering are rapidly being automated with various coding AI tools, as seen in the AI technology radar. Data engineering is one aspect where I see a few startups starting to disrupt.

Data Engineer 67

Data Engineer Data Engineering Engineering Kafka 67

Data Engineering Podcast

JANUARY 30, 2022

Matt Harrison is a Python expert with a long history of working with data who now spends his time on consulting and training. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. What are some of the utility features that you have found most helpful for data processing?

Data Engineer 100

Data Engineer Data Engineering Engineering Python 100

Striim

MARCH 21, 2025

To access real-time data, organizations are turning to stream processing. There are two main data processing paradigms: batch processing and stream processing. Your electric consumption is collected during a month and then processed and billed at the end of that period.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Data Engineering Weekly

FEBRUARY 2, 2025

Streamline code deployment, enhance collaboration, and ensure DevOps best practices with Astro's robust CI/CD capabilities. The evaluation process includes over 4,000 automated tests, measuring the percentage of passing unit tests, similarity to known passing states, and token usage. Automate Airflow deploys with built-in CI/CD.

Data Engineering 57

Data Engineering Data Engineer Engineering Data Lake 57

Christophe Blefari

JANUARY 14, 2023

On the data processing side there is Polars, a DataFrame library that could replace pandas. df = pl.read_csv("lost-objects-stations.csv", sep=";") Then you can use the same code as pandas to select the data (head, ["col"], etc.). With this release you can really mix Python and SQL code.

Python 130

Python Kafka Data Scala 130

Precisely

NOVEMBER 21, 2024

This relationship is particularly important in SAP® environments, where data and processes must work together seamlessly at scale. To achieve true transformation, you need an aligned approach where both processes and data management evolve together.

Data Integration 57

Data Integration High Quality Data Data Manufacturing 57

Engineering at Meta

FEBRUARY 4, 2025

To prevent this issue, we built verification in the post-processing stage to ensure that the user ID column in the data matches the identifier for the user whose logs we are generating. Finally, once content has been reviewed, it can be implemented in code using the renderers we described above.

Accessible 92

Accessible Accessibility Raw Data Data Warehouse 92

Engineering at Meta

AUGUST 27, 2024

Commonly, purpose limitation can rely on “point checking” controls at the point of data processing. This approach involves using simple if statements in code (“code assets”) or access control mechanisms for datasets (“data assets”) in data systems.

Programming Language 125

Programming Language Coding Data Warehouse Systems 125

Data Engineering Weekly

MARCH 23, 2025

Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA. The article highlights Nuage 3.0's

Data Engineer 62

Data Engineer Data Engineering Engineering Data 62

Snowflake

APRIL 9, 2025

KAWA combines analytics, automation and AI agents to help enterprises build data apps and AI workflows quickly and achieve their digital transformation goals. It connects structured and unstructured databases across sources and uses a no-code UI or Python for advanced and predictive analytics.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

databricks

APRIL 28, 2023

Databricks Delta Live Tables (DLT) radically simplifies the development of the robust data processing pipelines by decreasing the amount of code that data.

Coding 105

Coding Data Process Process Data 105

Ascend.io

DECEMBER 4, 2024

Examples include “reduce data processing time by 30%” or “minimize manual data entry errors by 50%.” It aims to streamline and automate data workflows, enhance collaboration and improve the agility of data teams. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. Hex Tech Logo]([link] Hex is a collaborative workspace for data science and analytics.

Building 130

Building Machine Learning SQL Python 130