Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Neither WhatsApp nor Secret Conversations operated in this manner, and we didn’t want all users to have to rely on a device-side storage system.

Building 145

Building Designing Consulting Accessibility 145

Confluent

MAY 23, 2019

She recounted a number of lessons Confluent has learned in building Confluent Cloud, and announced the availability of several new features in the cloud service. ?. We rightly spend a lot of time trying to figure out how to build things, so it was good to step back and see how our engineering work can drive internal cultural change as well.

Kafka 109

Kafka Cloud Database Engineering 109

Netflix Tech

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN. In 2019, Netflix moved thousands of container hosts to bare metal.

AWS 15

AWS Entertainment Software Engineer Cloud 15

Lyft Engineering

APRIL 3, 2023

This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders. Introduction: Scope of the Recommendation System The recommendation system covers user experiences throughout the ride journey.

Systems 90

Systems Pipeline-centric Machine Learning Transportation 90

Confluent

APRIL 16, 2019

To build the kinds of systems we are being called upon to build these days, we need infrastructure that gives equal priority to events and state together. and a couple of fantastic keynotes: Jay Kreps (CEO of Confluent and co-creator of Apache Kafka ® ) kept the unifying vision of the event streaming platform in front of us.

Kafka 102

Kafka Database Programming Systems 102

The Pragmatic Engineer

APRIL 6, 2023

This created an opportunity to build job sites which collect this data, make it easy to browse, and allow job seekers to apply to jobs paying at or above a certain level. For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing.

Software Engineer 231

Software Engineer Software Engineering Datasets Database 231

Netflix Tech

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. Refer to our recent overview for more details).

Metadata 88

Metadata Bytes Entertainment Data Mining 88

Confluent

OCTOBER 1, 2019

If you looked at the Kafka Summits I’ve been a part of as a sequence of immutable events (and they are, unless you know something about time I don’t), it would look like this: New York City 2017, San Francisco 2017, London 2018, San Francisco 2018, New York City 2019, London 2019, San Francisco 2019. Yes, you read that right.

Kafka 14

Kafka Database Cloud Systems 14

Cloudera

DECEMBER 1, 2023

Launched in 2019, this strategy aims to position the US as a leader in AI research, development, and deployment. It focuses on five key pillars: investing in research and development; unleashing government AI resources; setting standards and policy; building the AI workforce; and advancing trust and security. million), among others.

Building 109

Building Government Transportation Data Governance 109

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. 2019: Users can view their activity off Meta-technologies and clear their history. feature on Facebook. What are data logs?

Accessibility 92

Accessibility Accessible Raw Data Data Warehouse 92

Confluent

OCTOBER 16, 2019

As with any system out there, the data often needs processing before it can be used. In traditional data warehousing, we’d call this ETL, and whilst more “modern” systems might not recognise this term, it’s what most of us end up doing whether we call it pipelines or wrangling or engineering. Handling time.

Kafka 19

Kafka Building Data Coding 19

Netflix Tech

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. This insight led us to build Edgar: a distributed tracing infrastructure and user experience. Troubleshooting a session in Edgar When we started building Edgar four years ago, there were very few open-source distributed tracing systems that satisfied our needs.

Building 96

Building Transportation Java Metadata 96

Confluent

OCTOBER 9, 2019

Based on the votes of Summit attendees from within the Kafka Summit mobile app, here are the top-rated talks: Building Stream Processing Applications with Apache Kafka Using KSQL by Robin Moffatt of Confluent. With so many sessions to choose from, perhaps you’re wondering where to start. Why Stop the World When You Can Change It?

Kafka 17

Kafka Software Engineer Software Engineering Data Pipeline 17

The Pragmatic Engineer

MAY 1, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” AI dev tool startups need outlandish claims to grab attention.

Software Engineer 239

Software Engineer Software Engineering Programming Language Media 239

Confluent

SEPTEMBER 30, 2019

I even stopped by Build-A-Bear at lunchtime with the inaugural class of Confluent Community Catalysts ! He is the co-presenter of various O’Reilly training videos on topics ranging from Git to distributed systems, and is the author of Gradle Beyond the Basics. And, I saw fresh Kafka swag in the making. and all over the world.

Kafka 17

Kafka Architecture Cloud Designing 17

Netflix Tech

DECEMBER 10, 2019

Fall 2019 By Tom Richards , Carenina Garcia Motion , and Leslie Posada Hack Day at Netflix is an opportunity to build and show off a feature, tool, or quirky app. November 2019 was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Technology 52

Technology Project Systems Building 52

Netflix Tech

DECEMBER 10, 2019

Fall 2019 By Tom Richards , Carenina Garcia Motion , and Leslie Posada Hack Day at Netflix is an opportunity to build and show off a feature, tool, or quirky app. November 2019 was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Technology 52

Technology Project Systems Building 52

Data Engineering Podcast

DECEMBER 26, 2021

Zhamak Dehghani introduced the concepts behind this architectural patterns in 2019, and since then it has been gaining popularity with many companies adopting some version of it in their systems. How has your view of the principles of the data mesh changed since our conversation in July of 2019?

BI 130

BI Data Warehouse Data Engineering Data Engineer 130

Pinterest Engineering

JANUARY 4, 2024

In 2020, anticipating the growing needs of the business and to simplify our storage offerings, we decided to consolidate our different key-value systems in the company into a single unified service called KVStore. In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator.

Database 98

Database Building Datasets Relational Database 98

KDnuggets

AUGUST 15, 2019

While the process of building simple, domain-specific chatbots has gotten way easier, building large scale, multi-agent conversational applications remains a massive challenge.

Systems 66

Systems Building Engineering Process 66

Engineering at Meta

FEBRUARY 6, 2023

The app makes heavy use of code generation, spurred by Buck , our custom build system. Without heavy caching from our build system, engineers would have to spend an entire workday waiting for the app to build. If News Feed wanted to have a declarative UI, the team would have to build a new UI framework.

Architecture 145

Architecture Coding Engineering Systems 145

Cloudera

AUGUST 12, 2021

A 2020 retention report by the Work Institute revealed that over 42 million employees in the US left their jobs voluntarily in 2019, and this trend appeared to be increasing. This is especially important for organizations to set and meet diversity, equity and inclusion targets to build well-rounded and successful teams. .

Manufacturing 111

Manufacturing Insurance Consulting Education 111

Jesse Anderson

JANUARY 9, 2023

In 2019, Alibaba bought Ververica. This comparison seems like a clear point that Flink will be the system of choice. Streaming systems are already difficult enough and adding more complexity to the choice with lead to analysis paralysis. That’s because it was recently founded, but that doesn’t mean it wasn’t formidable.

Kafka 147

Kafka Technology Coding SQL 147

Lyft Engineering

JUNE 8, 2023

We’re looking for driven engineers to fortify our European operations and solve some of the hardest problems in building large distributed systems to support rideshare, mapping, and more. Lyft was founded in 2012 and went public in 2019, with the mission to improve people’s lives with the world’s best transportation.

Transportation 111

Transportation Algorithm Machine Learning Engineering 111

Confluent

MAY 30, 2019

Built with Prometheus and InfluxDB monitoring systems. To build an event streaming pipeline, Spring Cloud Data Flow provides a set of application types: A source represents the first step in the data pipeline, a producer that extracts data from the external systems like databases, filesystem, FTP servers, IoT devices, etc.

Kafka 95

Kafka Cloud Data Pipeline PostgreSQL 95

Edureka

FEBRUARY 12, 2025

Software Engineers, on the other hand, specialize in building and developing comprehensive systems, with an emphasis on architectural and engineering concepts. On the other hand, a Software Engineer focuses on specific areas of development, such as system design, algorithms, or a programming language.

Software Engineer 52

Software Engineer Software Engineering Computer Science Programming Language 52

Cloudera

MAY 27, 2021

An authoritarian regime is manipulating an artificial intelligence (AI) system to spy on technology users. When developing ethical AI systems, the most important part is intent and diligence in evaluating models on an ongoing basis,” said Santiago Giraldo Anduaga, director of product marketing, data engineering and ML at Cloudera.

Algorithm 119

Algorithm Media Consulting Machine Learning 119

Cloudera

DECEMBER 14, 2021

But with growing demands, there’s a more nuanced need for enterprise-scale machine learning solutions and better data management systems. They created a system to spread data across several servers with GPU-based processing so large datasets could be managed more effectively across the board. .

Transportation 111

Transportation Telecommunication Banking Data Lake 111

Knowledge Hut

OCTOBER 4, 2023

Below are the Power BI requirements for the system. Supported operating system: Power BI program can be installed in a device with the following operations systems. Windows Server 2019 Data Centre, server 2019 standard, server 2016 standard, server 2016 datacenter. GHz with the recommended level of 2.0

BI 52

BI Systems Raw Data Data Preparation 52

LinkedIn Engineering

OCTOBER 19, 2023

Over time, LinkedIn's engineering team expanded the stream processing ecosystem with more proprietary tools like Brooklin , facilitating data streaming across multiple stores and messaging systems, and Venice , serving as a storage system for ingesting batch and stream processing job outputs, among others.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Netflix Tech

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Logs and background requests are examples of this type of traffic.

Architecture 107

Architecture Systems Engineering Utilities 107

LinkedIn Engineering

APRIL 18, 2023

Minion (an agent on host) sees jobs and results by subscribing to events published on the event bus by master service, It uses ZMQ (ZeroMQ) to achieve high-speed, asynchronous communication between connected systems. Targeted minions execute the job on the host and return to master.

MySQL 103

MySQL Python Bytes Kafka 103

Netflix Tech

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN. We explore all the systems necessary to make and stream content from Netflix.

AWS 40

AWS Entertainment Software Engineer Software Engineering 40

Netflix Tech

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN. We explore all the systems necessary to make and stream content from Netflix.

AWS 40

AWS Entertainment Software Engineer Software Engineering 40

Confluent

JULY 3, 2019

Ingesting Twitter data is very easy with Kafka Connect , a framework for connecting Kafka with external systems. For more details on how to build a UD(A)F function, please refer to How to Build a UDF and/or UDAF in KSQL 5.0 Ingesting Twitter data. confluent-hub install jcustenborder/kafka-connect-twitter:latest.

Data Analysis 21

Data Analysis Kafka Datasets Java 21

DataKitchen

APRIL 13, 2021

Apache Oozie — An open-source workflow scheduler system to manage Apache Hadoop jobs. DBT (Data Build Tool) — A command-line tool that enables data analysts and engineers to transform data in their warehouse more effectively. Reflow — A system for incremental data processing in the cloud. Google Cloud Build . .

Consulting 121

Consulting Machine Learning Data Science Data Pipeline 121

Towards Data Science

FEBRUARY 15, 2023

The Data Lake architecture was proposed in a period of great growth in the data volume, especially in non-structured and semi-structured data, when traditional Data Warehouse systems start to become incapable of dealing with this demand. Let’s add the readings from 2019. Legend says, that this didn’t go well. The data became useless.

Data Lake 98

Data Lake Data Warehouse Hadoop Architecture 98