LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Pinterest Engineering

SEPTEMBER 19, 2024

Manager, Engineering | At Pinterest, we operate a large-scale online machine learning inference system, where feature caching plays a critical role to achieve optimal efficiency. Background Recommender systems are fundamental to Pinterest’s mission to inspire users to create a life they love.

Systems 64

Systems Utilities Architecture Software Engineer 64

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 138

Utilities Systems Architecture Coding 138

Netflix Tech

JUNE 13, 2023

This is where large-scale system migrations come into play. Our previous blog post presented replay traffic testing — a crucial instrument in our toolkit that allows us to implement these transformations with precision and reliability. Sticky Canary is an improvement to the traditional canary process that addresses this limitation.

Systems 114

Systems Utilities Entertainment Coding 114

The Pragmatic Engineer

AUGUST 10, 2023

The period from April to mid-May was challenging: I found myself in hiring freezes and canceled processes. Simplify helped me a lot to automatically fill applications in the most popular applicant tracking systems (ATS.) ’ How did you find the interview processes?

Engineering 195

Engineering Recruitment Retail Software Engineer 195

Data Engineering Weekly

SEPTEMBER 26, 2024

Last Friday, we discussed the challenges in bulk discovery and anonymization processes in data warehouses. Most cloud providers offer built-in encryption options and key management systems (KMS) , making it easier to stay compliant without sacrificing security. Once again, I want to thank the Data Heros community.

Systems 67

Systems Engineering Data Warehouse Architecture 67

Tweag

JUNE 7, 2023

The availability of deep learning frameworks like PyTorch or JAX has revolutionized array processing, regardless of whether one is working on machine learning tasks or other numerical algorithms. Nix is needed to simplify the installation and patching of system dependencies (not all our patches have been merged upstream yet).

Process 103

Process Coding Python Deep Learning 103

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Metadata Kafka Systems 139

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 100

Datasets Computer Science Systems Kafka 100

Cloudera

MAY 4, 2022

The availability and maturity of automated data collection and analysis systems is making it possible for businesses to implement AI across their entire operations to boost efficiency and agility. AI increasingly enables systems to operate autonomously, making self-corrections automatically as necessary. Process acceleration.

Insurance 130

Insurance Retail Finance Medical 130

Tweag

NOVEMBER 20, 2024

Two years ago I wrote a blog post to announce that the GHC wasm backend had been merged upstream. I’ve been too lazy to write another blog post about the project since then, but rest assured, the project hasn’t stagnated. If you take a look at htop , you’ll notice wasm32-wasi-ghc spawns a node child process.

Coding 137

Coding Programming Project Process 137

Engineering at Meta

OCTOBER 24, 2023

In our last blog post on automatic product deprecation , we talked about the complexities of product deprecations, and a solution Meta has built called the Systematic Code and Asset Removal Framework (SCARF). It leverages this analysis to submit change requests to remove this code from our systems.

Coding 139

Coding Programming Language Python MySQL 139

Robinhood

JUNE 6, 2024

Robinhood and Bitstamp customers can expect the same level of service, security and reliability and as we move forward, we are committed to maintaining transparency throughout this process. Robinhood expects the final deal consideration to be approximately $200 million in cash, subject to customary purchase price adjustments.

Retail 129

Retail Systems Process Management 129

Data Engineering Weekly

AUGUST 6, 2024

CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. Impacts source system performance during query execution.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Jesse Anderson

SEPTEMBER 14, 2023

that summarizes blog posts using LLMs. All the while, there will be a portion of the population that creates more and more complex systems with fewer people. Using LLMs to process unstructured data is amazing. This sort of work used to be the domain of data scientists specializing in NLP (natural language processing).

Data Engineering 147

Data Engineering Data Engineer Engineering NoSQL 147

Data Engineering Weekly

JULY 28, 2024

The blog is an excellent summarization of the common patterns emerging in GenAI platforms. Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. Canva writes about the evolution of experimentation system design and the current state of the function.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Podcast

FEBRUARY 26, 2023

In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. What are the technical systems that you are relying on to power the different data domains? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Machine Learning Metadata Data Integration 147

Maxime Beauchemin

APRIL 26, 2023

To support engineers in this process, I am introducing Promptimize , a Python toolkit designed to measure success and outcomes while doing prompt engineering. This blog post is about prompt engineering, its challenges, and the advantages of using Promptimize to support engineers building prompts. What a stellar assistant!

SQL 148

SQL Database Engineering Software Engineer 148

Data Engineering Weekly

JULY 21, 2024

The blog is an excellent summary of what one needs to know about Gen-AI to start. The current extraction models from unstructured data often rely on OCR and layout detection to process text and images separately, leading to potential indexing errors and inefficiencies. link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Edureka

FEBRUARY 8, 2023

In this blog we will get to know about the perks of ChatGPT for coding. This blog will help you learn how this effective tool can help you write code with ease, and we will also cover topics like: What is ChatGPT? Streamlining the implementation process for ChatGPT Fig.7 ” Fig.6 ChatGPT is designed to be simple to use.

Coding 130

Coding Deep Learning Programming Java 130

Data Engineering Weekly

JULY 14, 2024

The blog highlights the reasoning behind selecting dbt and Dagster and some of the key improvements while adopting them, such as handling race conditions in dbt incremental update and bulk backfilling with Dagster. Streaming execution to process a small chunk of data at a time. The buffer manager caches as many pages as possible.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow ecosystem ready and waiting with useful building blocks to accelerate the process.

Database 162

Database Technology Data Lake High Quality Data 162

Cloudera

APRIL 22, 2022

This blog post was written by Dean Bubley , industry analyst, as a guest author for Cloudera. . They are using disaggregated systems, plus distributed compute and data, for their own requirements — running their networks more efficiently, and dealing with customers and operations more flexibly. An example of the “networked cloud”.

Data Solutions 135

Data Solutions Amazon Web Services Data Storage Google Cloud 135

Engineering at Meta

OCTOBER 17, 2023

In this three-part blog series, we will discuss the complexities involved in removing a product from a complex portfolio of products and the framework Meta has built to drive the automation of this process, our Systematic Code and Asset Removal Framework (SCARF). different data types stored in 21 different data systems.

Coding 118

Coding Engineering Portfolio Data Schemas 118

Jesse Anderson

JUNE 12, 2024

We also discuss the various systems using Kafka’s protocol. When creating streaming systems, we can choose two of the three choices. I think it’s an interesting way of framing the tradeoffs we deal with in streaming systems. There was a chasm between the pub/sub-system and the data lake.

Kafka 147

Kafka Data Lake Architecture Cloud 147

DoorDash Engineering

MARCH 14, 2023

When dealing with failures in a microservice system, localized mitigation mechanisms like load shedding and circuit breakers have always been used, but they may not be as effective as a more globalized approach. Each section is accompanied by real outages drawn from our past blog posts that can be explored in greater detail.

Java 135

Java Metadata Database Systems 135

Christophe Blefari

FEBRUARY 24, 2023

This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. Finally Whatnot shared how the migrated their CD processes to ArgoCD and Pinterest now uses HTTP/3 , which I didn't even know it was existing. Is it Kafka?

Kafka 130

Kafka Data Lake Data Storage Data 130

Lyft Engineering

JUNE 6, 2023

This article will cover the following topics: Performance improvement process Strategies to profile streaming pipelines Common performance problems General guidelines to improve performance Performance Improvement Process The performance improvement of any software system is not an independent and isolated task but an iterative process.

Utilities 124

Utilities Coding Python Systems 124

Netflix Tech

SEPTEMBER 10, 2024

One issue that often complicates this process is the "noisy neighbor" problem. On Titus , our multi-tenant compute platform, a "noisy neighbor" refers to a container or system service that heavily utilizes the server's resources, causing performance degradation in adjacent containers.

Utilities 104

Utilities Coding Programming Java 104

Netflix Tech

MARCH 14, 2023

User provides a sample image to find other similar images Prior engineering work Approach #1: on-demand batch processing Our first approach to surface these innovations was a tool to trigger these algorithms on-demand and on a per-show basis. Processing took several hours to complete. Maintaining disparate systems posed a challenge.

Media 119

Media Building Algorithm Machine Learning 119

Christophe Blefari

JULY 13, 2024

AI News 🤖 OpenAI — Always the biggest news provider, announcements or dramas OpenAI was hacked, revealing internal secrets and raising national security concerns — The hacker reached OpenAI’s internal messaging systems early last year, stealing details of how OpenAI's technologies work from employees.

Kafka 130

Kafka AWS Data Database 130

Jesse Anderson

OCTOBER 5, 2023

In another post , I introduced a concept from Justin Coffey about how much systems can be changed from their original design. Teams must know this, or you’ll have vendor lock-in on an ostensibly open-source system. Flink’s batch processing is improving, and there are some caveats to know. million in cash and another $52.3

Kafka 195

Kafka Finance Cloud Designing 195

Engineering at Meta

AUGUST 27, 2024

At Meta, we’ve been diligently working to incorporate privacy into different systems of our software stack over the past few years. Purpose limitation, a core data protection principle, is about ensuring data is only processed for explicitly stated purposes.

Programming Language 118

Programming Language Coding Data Warehouse Systems 118

Pinterest Engineering

JANUARY 9, 2024

Ads and other content from businesses are an important part of the consumer journey because they add value to the discovery process. In this blog we will deep dive into some of our recent advancements in machine learning modeling to connect pinners with the most relevant ads.

Machine Learning 95

Machine Learning Architecture Engineering Utilities 95