DoorDash Engineering

FEBRUARY 22, 2023

In simple terms, “inventory” refers to the list of items present in a specific store of a Convenience and Grocery (CnG) merchant. The post How DoorDash Designed a Successful Write-Heavy Scalable and Reliable Inventory Platform appeared first on DoorDash Engineering Blog. Before we dive in, let’s define some important terminology.

Designing 125

Designing Database Architecture SQL 125

Netflix Tech

JANUARY 6, 2025

This situation presents both challenges and opportunities; while it may be more difficult to make initial progress, there are plenty of easy wins to capitalize on. These elements are critical for a titles eligibility in a row, accurate personalization, and an engaging presentation. artwork, trailers, supplemental messages).

Metadata 87

Metadata Algorithm Systems Building 87

Data Engineering Weekly

JANUARY 15, 2025

To address these challenges, AI Data Engineers have emerged as key players, designing scalable data workflows that fuel the next generation of AI systems. Complexity and Variability Each type of unstructured data—text, images, videos, or audio—presents unique challenges. Their role is not just important; it is essential.

Data Engineer 63

Data Engineer Data Engineering Unstructured Data Engineering 63

Pinterest Engineering

APRIL 4, 2025

In this blog, we will go through the technical design and share some offline and online results for our LLM-based search relevance pipeline. Technical Design LLM as Relevance Model Model Architecture We use a cross-encoder language model to predict a Pins relevance to a query, along with Pin text, as shown in Figure 1.

Machine Learning 56

Machine Learning Metadata Architecture Datasets 56

Monte Carlo

NOVEMBER 26, 2024

According to Tomasz, an AI use-case presents the opportunity for cost reduction if one of three criteria are met: 1. Like Google, large models are designed to service a variety of use-cases. While those numbers are certainly impressive by modern standards, they still present some key limitations for the use of AI.

Data Engineer 105

Data Engineer Data Engineering Engineering Unstructured Data 105

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. Obtaining this data from Hive in a format that can be presented to users is not straightforward. What are data logs?

Accessibility 80

Accessibility Accessible Raw Data Data Warehouse 80

Simon Späti

NOVEMBER 19, 2024

MVC is an interesting concept from the late 70s that separates the View (presentation) from the Controller via the Model. It has been used in designing web applications and is still heavily used, for example, in Ruby on Rails or Laravel, a popular PHP framework.

Designing 130

Designing IT Data 130

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. This complexity requires a mature data engineering team to design, implement, and manage it effectively.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Netflix Tech

DECEMBER 17, 2024

This article is the first in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. It is challenging to scale such bespoke solutions to ever-changing and increasingly complex businessneeds.

Engineering 84

Engineering Entertainment Amazon Web Services Utilities 84

Zalando Engineering

MARCH 6, 2025

Those without the capacity relied on an existing unified data source, such as our Presentation API, inheriting its version of product data. To serve the presentation view of a Product Offer, a multi-stage event-driven system merged Product, Price, and Stock events into a single structure. This led to competing sources of truth.

Algorithm 63

Algorithm Architecture Transportation Data Ingestion 63

Snowflake

JANUARY 16, 2025

Yet, the complexity and computational demands of LLM inference present a challenge. It is designed to facilitate research and prototype new ideas for post-training without getting overwhelmed by complex abstraction layers or generalizations. Inference costs remain prohibitive for many workloads.

Algorithm 97

Algorithm Data Analysis Building Process 97

Christophe Blefari

MARCH 1, 2023

This is a methodology that has been designed more than 20 years ago that optimise the storage used. In these articles there are a lot of cool presentations you should watch to understand deeper how dbt works. You can choose the strategy you want depending on your adapter (cf. examples on BigQuery ).

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Netflix Tech

JANUARY 2, 2025

This article is the second in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. To better guide the design and budgeting of future campaigns, we are developing an Incremental Return on Investment model. Need to catch up?

Engineering 76

Engineering Entertainment Designing Technology 76

The Pragmatic Engineer

OCTOBER 17, 2024

  This grant is designed to “support entrepreneurs, tech-geeks, developers, and socially engaged people, who are capable of challenging the way we search and discover information and resources on the internet” The team is tiny; only three people.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Waitingforcode

FEBRUARY 3, 2023

Message bus is a common architectural design in the Enterprise Design Patterns. But it's also present at a lower level to enable the event-driven behavior. Apache Spark is not an exception. It uses a publish/subscribe approach in various places.

Architecture 130

Architecture Designing IT 130

Engineering at Meta

JANUARY 27, 2023

The presenters Shrikrishna Khare and Srikrishna Gopu, talk about their experience designing, developing, and operating FBOSS: An in-house software built to manage and support a set of features required for data center switches of a large-scale Internet content provider.

Engineering 130

Engineering Software Engineering Software Engineer Transportation 130

The Pragmatic Engineer

AUGUST 22, 2023

Games Are Software, But in a Non-Standard Way Video games can often feel far removed from traditional software, given the modes of interaction and style of presentation are often unique to this space. I often explain this working relationship as that artists make it pretty , while designers and programmers make it work.

Software Engineering 235

Software Engineering Software Engineer Consulting Entertainment 235

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Cloudera

JANUARY 9, 2025

Simplifying to Amplify This renaming is part of a broader effort to simplify how we present our offerings. Best of all, they are all designed to work together seamlessly, providing you with the capabilities for a smooth path from raw data to AI-driven results. Within it, youll find capabilities that clearly map to what they deliver.

Machine Learning 85

Machine Learning Raw Data Government Algorithm 85

The Pragmatic Engineer

AUGUST 14, 2024

Below are experienced speakers who have presented at conferences and have confirmed they are open to such opportunities (including traveling to locations in the Netherlands.) See his talk at React Live 2023 (a case study of a successful migration) and at JSWORLD 2022 (designing high-performance React applications.) See more past talks.

Software Engineering 138

Software Engineering Software Engineer Education Architecture 138

Christophe Blefari

MAY 18, 2023

As I often do I've overlooked the 70 presentations and here a medley of what I've liked. The presentation gives another look at the semantic layer. During the demo, Llyod does some data analysis in the browser and it's just mind-blowing 🤯 At the same time someone Google also did a Calcite presentation.

Data 130

Data BI Consulting Data Science 130

Pinterest Engineering

JANUARY 31, 2025

Fig 1 illustrates a general multi-stage recommendation funnel design in Pinterest. General multi-stage recommendation system design in Pinterest. This section illustrates the current machine learning design of the two-tower machine learning model for learned retrieval at Pinterest.

Systems 67

Systems Metadata Machine Learning Architecture 67

Precisely

MARCH 18, 2025

There were many Gartner keynotes and analyst-led sessions that had titles like: Scale Data and Analytics on Your AI Journeys” What Everyone in D&A Needs to Know About (Generative) AI: The Foundations AI Governance: Design an Effective AI Governance Operating Model The advice offered during the event was relevant, valuable, and actionable.

Data Analytics 64

Data Analytics Data Governance Government Consulting 64

dbt Developer Hub

APRIL 20, 2025

The dbt MCP server presents our early explorations, but we anticipate that the Community will find many more. Determining the most useful tools for the dbt MCP What are the best and most useful set of tools to enable human in the loop and AI driven LLM access to structured data?

Structured Data 98

Structured Data SQL BI Project 98

The Pragmatic Engineer

AUGUST 14, 2024

Below are experienced speakers who have presented at conferences and have confirmed they are open to such opportunities (including traveling to locations in the Netherlands.) See his talk at React Live 2023 (a case study of a successful migration) and at JSWORLD 2022 (designing high-performance React applications.) See more past talks.

Software Engineering 130

Software Engineering Software Engineer Education Architecture 130

Cloudyard

DECEMBER 24, 2024

Read Time: 2 Minute, 11 Second In today’s data-driven world, organizations demand powerful tools to transform, analyze, and present their data seamlessly. GOLDEN Layer : Curated tables (dimensions and facts) designed for analytics and reporting. SILVER Layer : Cleansed and enriched data prepared for analytical processing.

Building 52

Building Raw Data Scala Business Intelligence 52

LinkedIn Engineering

OCTOBER 5, 2023

This focus on training pipeline steps which take in heterogeneous inputs with multiple objective functions, and emits co-trained models which are already part of an implicit inference graph centers the application development experience in the place where the AI engineer feels most at home: designing the recommender model itself.

IT 133

IT Metadata Designing Cloud 133

LinkedIn Engineering

APRIL 4, 2023

This blog post will provide an overview of how we approached metrics selection and design, system architecture and key product features. Metrics Design and Selection We started with a simple question: What should we measure? To answer that we followed a metrics design framework called “Goals-Signals-Metrics.” (GSM)

MySQL 122

MySQL Datasets Software Engineer Software Engineering 122

Data Engineering Weekly

FEBRUARY 18, 2025

Fluss is a streaming storage specifically designed for real-time analytics. Fluss and Kafka differ fundamentally in design principles. Kafka is designed for streaming events, but Fluss is designed for streaming analytics. Fluss is designed for real-time analytics. What is Fluss and its use cases?

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Data Engineering Podcast

MARCH 5, 2023

In terms of the conference, what are the factors that you consider when deciding how to group the different presentations into tracks or themes? What are the most interesting, unexpected, or challenging lessons that you have learned while working on selecting presentations for this year's event?

Building 147

Building Database Design Machine Learning Metadata 147

The Pragmatic Engineer

SEPTEMBER 28, 2023

I was still recovering from this failure in 2015, when a friend who designed the first version of Uber– Jelle Prins – pinged me, and said Uber was kickstarting an engineering office in Amsterdam. This last one was a great adventure; we went to the US to join the TechStars 2013 accelerator.

Software Engineer 314

Software Engineer Software Engineering Engineering Building 314

Data Engineering Weekly

MARCH 16, 2025

The blog narrates how Apache Arrow offers better data serialization efficiency and avoids design pitfalls from the past. The blog stresses the need for granular, structured feedback, especially from experts, and outlines key considerations for evaluation design. What we learned?

Data Engineer 58

Data Engineer Data Engineering Engineering Amazon Web Services 58

Netflix Tech

DECEMBER 17, 2024

How can we design systems that recognize these nuances and empower every title to shine and bring joy to ourmembers? They allow us to verify whether titles are presented as intended and investigate any discrepancies. Yet, these pages couldnt be more different. How do we bridge this gap?

Metadata 68

Metadata Systems Algorithm Data Analysis 68

Netflix Tech

MARCH 28, 2025

These insights have shaped the design of our foundation model, enabling a transition from maintaining numerous small, specialized models to building a scalable, efficient system. Addressing unique challenges, like cold start and presentation bias, the model also acknowledges the distinct differences between language tasks and recommendation.

Metadata 84

Metadata Bytes Data Mining Entertainment 84

Data Engineering Podcast

OCTOBER 22, 2023

With the broader audience comes the need to present data in a more approachable format. With the broader audience comes the need to present data in a more approachable format. Summary The primary application of data has moved beyond analytics. Can you describe what is encompassed by the idea of a data product strategy?

BI 162

BI SQL Machine Learning Programming Language 162

Snowflake

JUNE 20, 2024

Designed for processing large data sets, Spark has been a popular solution, yet it is one that can be challenging to manage, especially for users who are new to big data processing or distributed systems. The Snowpark Migration Accelerator builds an internal model representing the functionality present in the codebase.

Data Engineer 111

Data Engineer Data Engineering Scala Engineering 111