Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Lyft Engineering

APRIL 25, 2023

Building a large scale unsupervised model anomaly detection system — Part 2 Building ML Models with Observability at Scale By Rajeev Prabhakar , Han Wang , Anindya Saha Photo by Octavian Rosca on Unsplash In our previous blog we discussed the different challenges we faced for model monitoring and our strategy for addressing some of these problems.

Systems 76

Systems Building Machine Learning Datasets 76

Rockset

MARCH 28, 2024

Overview In this guide, you will: Gain a high-level understanding of vectors, embeddings, vector search, and vector databases, which will clarify the concepts we will build upon. Build a dynamic web application using vanilla CSS, HTML, JavaScript, and Flask, seamlessly integrating with the Rockset API and the OpenAI API.

Systems 52

Systems Building Database Utilities 52

Snowflake

JUNE 5, 2024

Enterprises can utilize gen AI to extract more value from their data and build conversational interfaces for customer and employee applications. It provides access to industry-leading large language models (LLMs), enabling users to easily build and deploy AI-powered applications. Adopting AI solutions can enhance overall efficiency.

Coding 89

Coding Building Management Government 89

Knowledge Hut

OCTOBER 29, 2023

Hospital Management System Type : Application development, Database management, Programming There is no shortage of computer science project topics out there. But if you are looking for something that's both technically challenging and socially relevant, consider a hospital management system. Source Code: Weather Forecast App 3.

Computer Science 111

Computer Science Coding Project Hospitality 111

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Why Postgres?

Database 130

Database Building PostgreSQL BI 130

Knowledge Hut

OCTOBER 27, 2023

Furthermore, project-based learning contributes to building a compelling portfolio that demonstrates your expertise and captivates potential employers. Android Local Train Ticketing System Developing an Android Local Train Ticketing System with Java, Android Studio, and SQLite. cvtColor(image, cv2.COLOR_BGR2GRAY) RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

Software Engineering 98

Software Engineering Software Engineer Coding Project 98

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Can you describe what Schemata is and the story behind it?

Systems 100

Systems Metadata Building MongoDB 100

Snowflake

AUGUST 25, 2023

Based on Snowflake’s testing, Meta’s newly released Code Llama models perform very well out-of-the-box. Code Llama models outperform Llama2 models by 11-30 percent-accuracy points on text-to-SQL tasks and come very close to GPT4 performance. We tested the out-of-the-box SQL performance of Code Llama before fine-tuning our own version.

Coding 75

Coding SQL Data Cleanse Database 75

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? Not only do you have to make it scalable and useful, but every architectural decision builds up over time. And building them yourself from scratch gives you full control of how you want them to behave. Building the data source.

Coding 53

Coding Cloud Storage Data Data Ingestion 53

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Pinterest Engineering

JANUARY 10, 2023

Liang Ma | Software Engineer, Core Eng; Wei Zhu | Software Engineer, Observability In early 2020, during a critical iOS out of memory incident (we have a blogpost for that), we realized that we didn’t have much visibility of how the app is running or a good system to look up for monitoring and troubleshooting.

Systems 57

Systems Building Software Engineering Software Engineer 57

Knowledge Hut

OCTOBER 26, 2023

IoT Project Ideas and Topics IoT is gaining traction by making daily life more efficient in areas like technologies that improve home comfort, traffic control, or environmental systems. Air Pollution Monitoring System Using IoT Air pollution has worsened almost all major cities' air quality. Source Code: Home Automation System 4.

Coding 98

Coding Project Electronics Cloud Computing 98

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. Modularity was key to enabling independent feature development and deployment across teams.

Building 68

Building Transportation Coding Designing 68

Knowledge Hut

DECEMBER 22, 2023

For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. Python interpreters are available on various operating systems, such as Windows , Linux, and Mac OS. But first, let us see what code editors and IDEs are. What is a Code Editor?

Python 97

Python Coding Programming Language Data Science 97

Engineering at Meta

DECEMBER 6, 2023

For example, we recently published an updated white paper, “ Meta’s Approach to Safer Private Messaging on Messenger and Instagram Direct Messaging ,” t hat sets out the industry-leading safety systems and tools available on Messenger. The post Building end-to-end security for Messenger appeared first on Engineering at Meta.

Building 144

Building Designing Consulting Coding 144

Monte Carlo

AUGUST 17, 2022

Over the years, I’ve helped companies of all sizes build and maintain data systems—from my days as a data engineer at Facebook to my current role as an end-to-end data solutions consultant. It’s never too early to start building reliability into your data system. The stakes also get higher.

Systems 52

Systems Building Data Data Engineer 52

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Knowledge Hut

OCTOBER 26, 2023

This will help you build a passion for coding as well as improve your programming skills. You can proceed by watching instructional videos to learn a skill or two, but you must finally begin building the projects. This project could also be a good way to get your feet wet with JS libraries/frameworks like React or Vue.

Coding 97

Coding Project Portfolio Building 97

Knowledge Hut

OCTOBER 29, 2023

Cloud-enabled attendance system We can use a cloud-enabled automatic attendance system to scan details. Administrators must register new students/employees on the system and provide some personal information. The entire system is powered by electricity. This system will also contain patient and contact information.

Cloud Computing 98

Cloud Computing Coding Cloud Project 98

Monte Carlo

OCTOBER 7, 2024

Monte Carlo, the data observability leader, today announced industry-first features that help identify Databricks query code changes and Gitlab pull requests that result in data quality incidents. PRs–along with associated author, date, and link information– are overlaid across table freshness and volume charts.

Coding 52

Coding Kafka Data Warehouse AWS 52

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Knowledge Hut

APRIL 24, 2024

Furthermore, project-based learning contributes to building a compelling portfolio that demonstrates your expertise and captivates potential employers. Android Local Train Ticketing System Developing an Android Local Train Ticketing System with Java, Android Studio, and SQLite. cvtColor(image, cv2.COLOR_BGR2GRAY) RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

Software Engineering 52

Software Engineering Software Engineer Coding Project 52

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Tweag

JULY 26, 2023

The vast majority of the Rust projects are using Cargo as a build tool. Cargo is great when you are developing and packaging a single Rust library or application, but when it comes to a fast-growing and complex workspace, one could be attracted to the idea of using a more flexible and scalable build system.

Building 97

Building Metadata Coding Project 97

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. And don’t forget to thank them for their continued support of this show!

Systems 100

Systems Software Engineering Software Engineer Data Warehouse 100

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Knowledge Hut

JUNE 26, 2023

Top Data Engineering Projects with Source Code Data engineers make unprocessed data accessible and functional for other data professionals. Source Code: Stock and Twitter Data Extraction Using Python, Kafka, and Spark 2. Source Code: Extracting Inflation Rates from CommonCrawl and Building a Model B.

Data Engineer 98

Data Engineer Data Engineering Coding Project 98

The Pragmatic Engineer

MARCH 12, 2024

” Brooks agrees with this observation, and suggests a radical solution: have as few senior programmers as possible, and build a team around each one – a bit like how a hospital surgeon leads a whole team. CI/CD : running automated tests on all changes, and deploying code to production automatically. The toolsmith.

Engineering 269

Engineering Programming Language Hospitality Programming 269

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support.

Cloud 273

Cloud Metadata AWS Cloud Computing 273

Rock the JVM

JUNE 13, 2024

A practical guide to building a distributed system with Scala and Pekko: learn how to run other people's code in a real-world scenario

Scala 52

Scala Coding Engineering Systems 52

Knowledge Hut

JUNE 9, 2024

Systems engineering is a diverse field of information technology engineering and engineering management that focuses on the design and administration of composite systems throughout their life cycles. Let's explore what is system engineer? They will manage many teams, testing, and product or system development analysis.

Systems 52

Systems Engineering Telecommunication Electronics 52

Data Engineering Podcast

FEBRUARY 1, 2021

As the systems that we build continue to grow in scale and complexity the need for reliable and manageable monitoring platforms increases proportionately. In this episode Rob Skillington, CTO of Chronosphere, shares his experiences building metrics systems that provide observability to companies that are operating at extreme scale.

Systems 100

Systems Cloud BI Data Warehouse 100

Knowledge Hut

OCTOBER 27, 2023

On an unclean and disorganised dataset, it is impossible to build an effective and solid model. Data Science Projects for Beginners with code For students new to Python or data science, we   will provide a list of data science project ideas. The data science projects for beginners with source code link to GitHub repo are listed below.

Data Science 96

Data Science Coding Project Datasets 96

Data Engineering Podcast

NOVEMBER 30, 2020

However, despite all of the tools and managed distributions of those streaming engines it is still a challenge to build a robust and reliable pipeline for streaming data integration, especially if you need to expose those capabilities to non-engineers. And don’t forget to thank them for their continued support of this show!

Data Integration 100

Data Integration Coding BI Data Engineer 100