Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

JANUARY 21, 2019

Summary Building internal expertise around big data in a large organization is a major competitive advantage. However, it can be a difficult process due to compliance needs and the need to scale globally on day one. What was the transition process like, migrating data silos into a uniformly managed platform?

Big Data 100

Big Data Systems Building Hadoop 100

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 93

Systems Media Machine Learning Data Warehouse 93

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. Can you start by describing what the Jepsen project is?

Systems 100

Systems Building Scala Java 100

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. What are the experimental methods that you are using to gain understanding in the opportunities and practical limits of those systems? What do you have planned for the future of your academic research?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Netflix Tech

FEBRUARY 18, 2022

To this end, we developed a Rapid Event Notification System (RENO) to support use cases that require server initiated communication with devices in a scalable and extensible manner. In this blog post, we will give an overview of the Rapid Event Notification System at Netflix and share some of the learnings we gained along the way.

Systems 133

Systems Architecture Portfolio Designing 133

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Rockset

MARCH 28, 2024

Overview In this guide, you will: Gain a high-level understanding of vectors, embeddings, vector search, and vector databases, which will clarify the concepts we will build upon. Build a dynamic web application using vanilla CSS, HTML, JavaScript, and Flask, seamlessly integrating with the Rockset API and the OpenAI API.

Systems 52

Systems Building Database Utilities 52

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Precisely

NOVEMBER 7, 2024

Key Takeaways: In the face of ransomware attacks, a resilience strategy for IBM i systems must include measures for prevention, detection, and recovery. No platform is immune, not even the reliable and secure IBM i systems. So, how can you keep your IBM i systems resilient even as ransomware risks are on the rise?

Systems 59

Systems Accessible Accessibility Programming 59

Knowledge Hut

JUNE 4, 2024

The Nodejs Commander package is the first choice to build command-line interface Node CLI as it offers many features to design CLI. Learn more about building CLI by Node.js The shebang line is used for Linux or UNIX type systems, but the node requires it for Windows and macOS, too, for proper installation and execution of the script.

Process 52

Process Building Utilities Programming 52

Monte Carlo

AUGUST 17, 2022

Over the years, I’ve helped companies of all sizes build and maintain data systems—from my days as a data engineer at Facebook to my current role as an end-to-end data solutions consultant. It’s never too early to start building reliability into your data system. The stakes also get higher.

Systems 52

Systems Building Data Data Engineer 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Can you describe what Schemata is and the story behind it?

Systems 100

Systems Metadata Building MongoDB 100

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. We feel your pain. It ends up being anything but that.

Building 147

Building Machine Learning Metadata Data Integration 147

Netflix Tech

JANUARY 10, 2024

Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. The Netflix video processing pipeline went live with the launch of our streaming service in 2007. The Netflix video processing pipeline went live with the launch of our streaming service in 2007.

Process 93

Process Pipeline-centric Media Metadata 93

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Knowledge Hut

DECEMBER 28, 2023

The information system is a very vast concept that encompasses several aspects like database management, the communication system, various devices, several connections, the internet, collection, organization, and storing data and other information-related applications that are typically used in a business forum.

Systems 52

Systems Telecommunication Technology Certification 52

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Pinterest Engineering

JANUARY 10, 2023

Liang Ma | Software Engineer, Core Eng; Wei Zhu | Software Engineer, Observability In early 2020, during a critical iOS out of memory incident (we have a blogpost for that), we realized that we didn’t have much visibility of how the app is running or a good system to look up for monitoring and troubleshooting.

Systems 57

Systems Building Software Engineering Software Engineer 57

AltexSoft

DECEMBER 23, 2022

Whether your goal is data analytics or machine learning , success relies on what data pipelines you build and how you do it. Integrating data from numerous, disjointed sources and processing it to provide context provides both opportunities and challenges. Order of process phases. Data engineering in 14 minutes. What is ELT?

Process 52

Process Building Raw Data Data Lake 52

LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Why Postgres? Rudderstack]([link] Introducing RudderStack Profiles.

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

DECEMBER 30, 2019

In order to support their customers, they need to capture, process, and analyze massive amounts of timeseries data with a high degree of uptime and reliability. What are some of the complexities and challenges that you face in your work as a result of the volume of data that you are processing?

Process 100

Process Building Hadoop Java 100

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Engineering at Meta

AUGUST 9, 2023

Explore is one of the largest recommendation systems on Instagram. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and flexible. locally popular media), which further contributes to system scalability.

Systems 92

Systems Media Algorithm Machine Learning 92

Netflix Tech

MARCH 14, 2023

We build creator tooling to enable these colleagues to focus their time and energy on creativity. User provides a sample image to find other similar images Prior engineering work Approach #1: on-demand batch processing Our first approach to surface these innovations was a tool to trigger these algorithms on-demand and on a per-show basis.

Media 118

Media Building Algorithm Machine Learning 118

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. We need a systemized source of truth that can simulate various performance factors across compute, storage, and network collectively. For instance, the AI Research SuperCluster for AI research.

Systems 105

Systems Architecture Designing Utilities 105

Engineering at Meta

DECEMBER 6, 2023

Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration. Neither WhatsApp nor Secret Conversations operated in this manner, and we didn’t want all users to have to rely on a device-side storage system.

Building 144

Building Designing Consulting Coding 144

Pinterest Engineering

SEPTEMBER 12, 2023

Behind the scenes, hundreds of ML engineers iteratively improve a wide range of recommendation engines that power Pinterest, processing petabytes of data and training thousands of models using hundreds of GPUs. As model architecture building blocks (e.g. It often requires a long process that touches many languages and frameworks.

Data Process 102

Data Process Process Datasets Software Engineering 102

Knowledge Hut

JANUARY 28, 2024

The change control process is a crucial aspect of project management intended to manage and regulate changes made to the project plan, schedule, and budget. These change control process steps are planning, analyzing, approval, testing, implementing, and closing. The change request kickstarts the process of change control.

Process 98

Process Project Designing Pharmaceutical 98

Knowledge Hut

MAY 17, 2024

In the early 1940s, Toyota automotive in Japan used a simple planning system to manage and control work and inventory at each stage optimally. It is related to lean and just-in-time JIT production, where it is used as a scheduling system to indicate what to produce, when, and how much. How does Kanban System work?

Systems 52

Systems Manufacturing Electronics Certification 52

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. The disconnection between the operational teams immersed in the day-to-day functions and those extracting business value from data generated in the operational processes still remains a significant friction point.

Systems 78

Systems Raw Data Metadata Data Cleanse 78