Pinterest Engineering

JANUARY 31, 2025

Modern large-scale recommendation systems usually include multiple stages where retrieval aims at retrieving candidates from billions of candidate pools, and ranking predicts which item a user tends to engage from the trimmed candidate set retrieved from early stages [2]. General multi-stage recommendation system design in Pinterest.

Systems 67

Systems Metadata Machine Learning Architecture 67

Striim

JANUARY 30, 2025

In this post, well explore how real-time data and AI-driven analytics reshape crisis management across industries such as healthcare, logistics, and emergency services. The Power of Real-Time Data in Crisis Management When a crisis unfolds, data moves at lightning speed.

Systems 52

Systems Management Hospitality Healthcare 52

Snowflake

FEBRUARY 13, 2025

. "Serverless computing" has enabled customers to use cloud capabilities without provisioning, deploying and managing either hardware or software resources. Snowflake has embraced serverless since our founding in 2012, with customers providing their code to load, manage and query data and us taking care of the rest.

Management 85

Management Government Cloud Unstructured Data 85

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Precisely

FEBRUARY 6, 2025

The traditional ways of operations management are over modernization and holistic approaches are now essential. Success in tackling modernization of IT operations management starts with assessing where your team is. Delivering data from IBM systems on a delay to ITOps platforms is a recipe for service disruptions. Whats next?

IT 63

IT Management Datasets Systems 63

KDnuggets

MAY 14, 2025

In this article, you'll master 10 essential Linux file system commands. This guide provides helpful examples to make working with files easier.

Systems 66

Systems Data Management Management Data 66

The Pragmatic Engineer

SEPTEMBER 19, 2024

Without the backing of management, a large-scale rewrite is likely to fail. In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. By doing so, I got to see every screen of the system.

Recruitment 297

Recruitment Java Coding Project 297

Precisely

FEBRUARY 26, 2025

Lets take a closer look at these exciting innovations and explore how theyll help you tackle six top data management challenges. Automated metadata management – AI-generated catalog asset descriptions significantly reduce manual efforts and improve metadata quality – enabling teams to focus on more strategic tasks.

Data Integration 63

Data Integration Data Management Management Data Governance 63

Data Engineering Podcast

MAY 12, 2024

Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior. Summary Building a data platform is a substrantial engineering endeavor.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Monte Carlo

MARCH 28, 2025

In this article, Ill share how even the best AI applications can break, and share how leading teams are managing reliability at scale across the ever-evolving data + AI estate. System Data + AI applications rely on a complex and interconnected web of tools and systems to deliver insights, models and automations.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

Precisely

JANUARY 9, 2025

In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s. There was no easy way to consolidate and analyze this data to more effectively manage our business. But simply moving the data wasnt enough.

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

RandomTrees

DECEMBER 4, 2024

As one of the most important sectors of the global economy, the food and beverage (F&B) industry works in highly volatile conditions and ensures its success by reducing waste and managing inventories. Managing production and consumption, meeting deadlines, cutting waste, and being environmentally friendly are always a challenge.

Food 52

Food Management Manufacturing Machine Learning 52

Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

databricks

DECEMBER 23, 2024

We are excited to announce that Gartner has recognized Databricks as a Leader for a fourth consecutive year in the 2024 Gartner Magic.

Database 110

Database Systems Cloud Management 110

The Pragmatic Engineer

NOVEMBER 21, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications. 

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Jesse Anderson

MAY 5, 2025

He details managing large, cross-functional data teams using a hub-and-spoke model and the importance of fostering high-integrity leadership. Join us for a technical discussion covering experimentation, data platform architecture, observability, the practicalities of scaling data teams, and the future of reliable AI systems.

Unstructured Data 130

Unstructured Data Finance Metadata Architecture 130

KDnuggets

JANUARY 22, 2025

Are you looking for an easier way to manage files across different storage systems? fsspec is a Python library that simplifies file handling by providing a unified interface for file management.

Python 113

Python Management Project Systems 113

The Pragmatic Engineer

MAY 12, 2023

OpsGenie is Atlassian's incident management tool, which is widespread thanks to distribution. Incident management. At incident.io, we believe that we’re the most sensible incident response and management tool for companies looking to do more than just alert. However, there is little awareness of these.

Systems 284

Systems Management Engineering IT 284

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products.

Data Warehouse 115

Data Warehouse SQL Programming Language Data 115

Snowflake

JANUARY 8, 2025

AI is both a contributor to the problem more data to secure, more attack surface and a potential boon, providing tools to manage amounts of data that humans cant grasp on their own. Compared to the traditional security incident and event management tools, security data lakes are generally more flexible, scalable and cost effective.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

Engineering at Meta

APRIL 28, 2025

Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale.

Metadata 79

Metadata Data Utilities Data Warehouse 79

Netflix Tech

NOVEMBER 12, 2024

Failures in a distributed system are a given, and having the ability to safely retry requests enhances the reliability of the service. Implementing idempotency would likely require using an external system for such keys, which can further degrade performance or cause race conditions.

Datasets 101

Datasets Computer Science Systems Kafka 101

The Pragmatic Engineer

FEBRUARY 29, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. But first, a few current cases of systems whose developers didn’t: In Sweden, card payments are down at a leading supermarket chain. Subscribe to get issues like this in your inbox, every week.

Software Engineer 281

Software Engineer Software Engineering Banking Engineering 281

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

The Pragmatic Engineer

OCTOBER 17, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Internal comms: Chat: Slack Coordination / project management: Linear 3. Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. To get full issues twice a week, subscribe here.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

dbt Developer Hub

APRIL 20, 2025

Both AI agents and business stakeholders will then operate on top of LLM-driven systems hydrated by the dbt MCP context. Todays system is not a full realization of the vision in the posts shared above, but it is a meaningful step towards safely integrating your structured enterprise data into AI workflows. Why does this matter?

Structured Data 98

Structured Data SQL BI Project 98

The Pragmatic Engineer

MARCH 12, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. This is how Brooks put it: “Programming managers have long recognized wide productivity variations between good programmers and poor ones. (.)

Engineering 329

Engineering Programming Language Hospitality Programming 329

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. But leveraging a supposedly neutral platform (the WordPress plugin manager) should not be the way to win in business – at least not in open source.

Engineering 357

Engineering Government Project AWS 357

The Pragmatic Engineer

APRIL 11, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Sella is a mid-sized bank in Italy with around $14B in assets under management. Still, I’m puzzled by how long the system has been down. To get full issues twice a week, subscribe here.

Banking 271

Banking Utilities Database Engineering 271

The Pragmatic Engineer

SEPTEMBER 22, 2023

The npm or pnpm package managers do this additional check and so are slower, though arguably more reliable. For most commercial open source companies, the approach tends to be to offer a managed, dedicated service of the open source software, or to provide additional features on top of it.

Programming Language 316

Programming Language Project Coding Engineering 316

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! Automattic generates most of its revenue by offering managed Wordpress hosting. WP Engine also sells managed Wordpress services, making it a direct competitor to Automattic.

Consulting 347

Consulting AWS Engineering Software Engineering 347

Christophe Blefari

JANUARY 11, 2025

AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. Meaning: a YAML configuration system for ingestion and transformations, and now, visualisation with BI-as-code. Meanwhile, the AI landscape remains unpredictable.

Data 130

Data Data Warehouse Coding Programming Language 130

The Pragmatic Engineer

OCTOBER 10, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. For some reason, Amazon didn’t have any senior engineers on the part of the EC2 team that was responsible for the EC2 instance management, and for the Windows business. It was a lot of fun!

Software Engineering 327

Software Engineering Software Engineer Architecture Media 327

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. It is meant to handle massive volumes of data across many commodity servers while maintaining high availability with no single point of failure.

NoSQL 259

NoSQL Database Project Systems 259

The Pragmatic Engineer

MAY 11, 2023

A very popular open-source solution for systems and services monitoring. Prometheus can be self-hosted, but several cloud providers also offer managed Prometheus services: both Google Cloud and AWS have this service in production, while Azure has it in preview. Source: Grafana.org Clickhouse : log management.

AWS 347

AWS Software Engineer Software Engineering Google Cloud 347

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. This blog post is the second in a three-part series on migrations.

Data Warehouse 104

Data Warehouse Cloud PostgreSQL Hadoop 104