LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Towards Data Science

APRIL 30, 2024

Balancing correctness, latency, and cost in unbounded data processing Image created by the author. Intro Google Dataflow is a fully managed data processing service that provides serverless unified stream and batch data processing. It is the first choice Google would recommend when dealing with a stream processing workload.

Google Cloud 92

Google Cloud Process Cloud Lambda Architecture 92

Snowflake

NOVEMBER 26, 2024

This blog post is the second in a three-part series on migrations. A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Striim

MARCH 21, 2025

What is Real-Time Stream Processing? To access real-time data, organizations are turning to stream processing. To access real-time data, organizations are turning to stream processing. There are two main data processing paradigms: batch processing and stream processing.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. Meanwhile, the AI landscape remains unpredictable.

Data 130

Data Data Warehouse Coding Programming Language 130

Striim

JANUARY 30, 2025

When integrated effectively, AI and machine learning (ML) models can process data streams at near-zero latency, empowering teams to make split-second decisions. Systems must be capable of handling high-velocity data without bottlenecks. Thats where real-time artificial intelligence (AI) can help.

Systems 52

Systems Management Hospitality Healthcare 52

Cloudera

NOVEMBER 13, 2024

Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches. This guarantees data quality and automates the laborious, manual processes required to maintain data reliability.

Metadata 90

Metadata Management Data Governance Government 90

Netflix Tech

APRIL 8, 2025

By Cheng Xie , Bryan Shultz , and Christine Xu In a previous blog post , we described how Netflix uses eBPF to capture TCP flow logs at scale for enhanced network insights. Delays and failures are inevitable in distributed systems, which may delay IP address change events from reaching FlowCollector. With 30 c7i.2xlarge

AWS 75

AWS Kafka Cloud Programming 75

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 98

Systems Machine Learning Data Mining Algorithm 98

Data Engineering Weekly

OCTOBER 27, 2024

Astasia Myers: The three components of the unstructured data stack LLMs and vector databases significantly improved the ability to process and understand unstructured data. The blog is an excellent summary of the existing unstructured data landscape. The blog from Meta discusses how it designed a privacy-preserving storage.

Data Engineer 58

Data Engineer Data Engineering Engineering Unstructured Data 58

Jesse Anderson

FEBRUARY 11, 2025

Semih is a researcher and entrepreneur with a background in distributed systems and databases. He then pursued his doctoral studies at Stanford University, delving into the complexities of database systems. Dont forget to subscribe to my YouTube channel to get the latest on Unapologetically Technical!

Computer Science 130

Computer Science Database Design Software Engineering Software Engineer 130

Edureka

APRIL 22, 2025

This blog will explore the significant advancements, challenges, and opportunities impacting data engineering in 2025, highlighting the increasing importance for companies to stay updated. In 2025, this blog will discuss the most important data engineering trends, problems, and opportunities that companies should be aware of.

Data Engineer 52

Data Engineer Data Engineering Engineering Consulting 52

Engineering at Meta

APRIL 28, 2025

Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process.

Metadata 64

Metadata Data Utilities Data Warehouse 64

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Utilities Metadata 66

DataKitchen

DECEMBER 9, 2024

This blog dives into the remarkable journey of a data team that achieved unparalleled efficiency using DataOps principles and software that transformed their analytics and data teams into a hyper-efficient powerhouse. Starting simply and iterating quickly gave the team time to build foundational processes before adding complexity and scaling.

Pharmaceutical 64

Pharmaceutical Data Lake Cloud Storage Project 64

Data Engineering Weekly

APRIL 20, 2025

The blog took out the last edition’s recommendation on AI and summarized the current state of AI adoption in enterprises. One of the core challenges of data engineering, as the author put it elegantly, The core difficulty lies in the fact that each step in the process requires specialized domain knowledge.

Data Engineer 68

Data Engineer Data Engineering Engineering Kafka 68

Pinterest Engineering

NOVEMBER 18, 2024

Liang Mou; Staff Software Engineer, Logging Platform | Elizabeth (Vi) Nguyen; Software Engineer I, Logging Platform | In today’s data-driven world, businesses need to process and analyze data in real-time to make informed decisions. Real-Time Data Processing : CDC enables real-time data processing by capturing changes as they happen.

Kafka 50

Kafka MySQL Database Software Engineer 50

The Pragmatic Engineer

AUGUST 10, 2023

The period from April to mid-May was challenging: I found myself in hiring freezes and canceled processes. Simplify helped me a lot to automatically fill applications in the most popular applicant tracking systems (ATS.) ’ How did you find the interview processes?

Engineering 227

Engineering Recruitment Retail Software Engineer 227

Snowflake

NOVEMBER 11, 2024

Here’s how Snowflake Cortex AI and Snowflake ML are accelerating the delivery of trusted AI solutions for the most critical generative AI applications: Natural language processing (NLP) for data pipelines: Large language models (LLMs) have a transformative potential, but they often batch inference integration into pipelines, which can be cumbersome.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Data Engineering Weekly

MARCH 2, 2025

I found the blog to be a fresh take on the skill in demand by layoff datasets. DeepSeek continues to impact the Data and AI landscape with its recent open-source tools, such as Fire-Flyer File System (3FS) and smallpond. The blog provides an excellent analysis of smallpond compared to Spark and Daft.

Data Engineer 63

Data Engineer Data Engineering Engineering Datasets 63

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 138

Utilities Systems Architecture Coding 138

Tweag

JUNE 7, 2023

The availability of deep learning frameworks like PyTorch or JAX has revolutionized array processing, regardless of whether one is working on machine learning tasks or other numerical algorithms. Nix is needed to simplify the installation and patching of system dependencies (not all our patches have been merged upstream yet).

Process 114

Process Coding Python Deep Learning 114

Picnic Engineering

APRIL 7, 2025

A little over a year ago, we shared a blog post about our journey to enhance customers meal planning experience with personalized recipe recommendations. We explained how a system that learns from your tastes and habits could solve this issue, ultimately making the daily task of choosing meals both effortless and inspiring.

Datasets 52

Datasets Systems Architecture Machine Learning 52

Cloudyard

DECEMBER 24, 2024

In this blog, well explore Building an ETL Pipeline with Snowpark by simulating a scenario where commerce data flows through distinct data layersRAW, SILVER, and GOLDEN.These tables form the foundation for insightful analytics and robust business intelligence. SILVER Layer : Cleansed and enriched data prepared for analytical processing.

Building 52

Building Raw Data Scala Business Intelligence 52

Pinterest Engineering

APRIL 7, 2025

In recent years, while managing Pinterests EC2 infrastructure, particularly for our essential online storage systems, we identified a significant challenge: the lack of clear insights into EC2s network performance and its direct impact on our applications reliability and performance. 4xl with up to 12.5

AWS 57

AWS Bytes Database Data Ingestion 57

Data Engineering Weekly

FEBRUARY 23, 2025

It covers nine categories: storage systems, data lake platforms, processing, integration, orchestration, infrastructure, ML/AI, metadata management, and analytics. I found the blog to be a comprehensive roadmap for data engineering in 2025. Let me know in the comments. link] All rights reserved ProtoGrowth Inc, India.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Netflix Tech

JANUARY 2, 2025

However, due to the absence of a control group in these countries, we adopt a synthetic control framework ( blog post ) to estimate the counterfactual scenario. Each format has a different production process and different patterns of cash spend, called our Content Forecast. As plans change, the cash forecast will change.

Engineering 76

Engineering Entertainment Designing Technology 76

DataKitchen

NOVEMBER 5, 2024

The Medallion architecture is a design pattern that helps data teams organize data processing and storage into three distinct layers, often called Bronze, Silver, and Gold. This foundational layer is a repository for various data types, from transaction logs and sensor data to social media feeds and system logs.

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Data Engineering Weekly

FEBRUARY 9, 2025

[link] QuantumBlack: Solving data quality for gen AI applications Unstructured data processing is a top priority for enterprises that want to harness the power of GenAI. It brings challenges in data processing and quality, but what data quality means in unstructured data is a top question for every organization.

Data Engineer 58

Data Engineer Data Engineering Engineering Unstructured Data 58

Data Engineering Weekly

APRIL 6, 2025

The article summarizes the recent macro trends in AI and data engineering, focusing on Vibe coding, human-in-the-loop system design, and rapid simplification of developer tooling. Unlike coding, we never (or rarely) apply a code review process for documentation. The Grab blog delights me since I have tried to do this many times.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Cloudyard

APRIL 22, 2025

Read Time: 2 Minute, 34 Second Introduction In modern data pipelines, especially in cloud data platforms like Snowflake, data ingestion from external systems such as AWS S3 is common. In this blog, we introduce a Snowpark-powered Data Validation Framework that: Dynamically reads data files (CSV) from an S3 stage.

Data Validation 52

Data Validation Data Ingestion Data Pipeline AWS 52

Netflix Tech

JUNE 13, 2023

This is where large-scale system migrations come into play. Our previous blog post presented replay traffic testing — a crucial instrument in our toolkit that allows us to implement these transformations with precision and reliability. Sticky Canary is an improvement to the traditional canary process that addresses this limitation.

Systems 114

Systems Utilities Entertainment Coding 114

dbt Developer Hub

JANUARY 22, 2025

By building on top of tools that truly understand SQL, it is possible to create systems that are much more capable, resilient and flexible than weve seen to date. At Level 2, the system produces a complete Logical Plan. Thats hard to demonstrate in a blog post, but fortunately theres another easier option: look at some failing queries.

SQL 78

SQL Database Coding Technology 78

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. It's good to know about Dapr and restate.dev.

Data Engineer 63

Data Engineer Data Engineering Engineering Pipeline-centric 63

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Scott Logic

JANUARY 13, 2025

This blog post will highlight the experience I had as a relatively junior developer on a Scott Logic project. The aim was to build a system that would allow them to compare their forecasted Air Quality Index (AQI) values with actual measured AQI data. We use powerful Python libraries such as Pandas and Xarray for this task.

Python 52

Python Project Coding Database 52

Robinhood

JUNE 6, 2024

Robinhood and Bitstamp customers can expect the same level of service, security and reliability and as we move forward, we are committed to maintaining transparency throughout this process. Robinhood expects the final deal consideration to be approximately $200 million in cash, subject to customary purchase price adjustments.

Retail 137

Retail Systems Management Process 137