Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

KDnuggets

NOVEMBER 6, 2024

Build an LLM application by easily picking and dropping components and connecting them, such as a vector store, web search, memory, and custom prompt.

Coding 152

Coding Building 152

The Pragmatic Engineer

MARCH 12, 2024

” Brooks agrees with this observation, and suggests a radical solution: have as few senior programmers as possible, and build a team around each one – a bit like how a hospital surgeon leads a whole team. CI/CD : running automated tests on all changes, and deploying code to production automatically. The toolsmith.

Engineering 274

Engineering Programming Language Hospitality Programming 274

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. fix the code # fix code 7.

Systems 141

Systems Building Java Programming Language 141

KDnuggets

AUGUST 28, 2024

Learn how to build and deploy an end-to-end data pipeline using Prefect with a few lines of code.

Data Pipeline 120

Data Pipeline Building Data Coding 120

The Pragmatic Engineer

JUNE 1, 2023

Instead of starting with coding, Juraj kicked off by sketching. He sketched out what he wanted the final product to look like: The sketch Juraj made, before starting any coding And he sketched how he envisioned the observability part to work: The sketch of the monitoring system Phase 1: Infrastructure (October-November).

Education 363

Education Project PostgreSQL Software Engineer 363

Analytics Vidhya

SEPTEMBER 12, 2024

Picture this: thus, unlike many other extensions that require deep setup and constant coding, […] The post Setup Mage AI with Postgres to Build and Manage Your Data Pipeline appeared first on Analytics Vidhya. That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 144

Data Pipeline Building Management Data 144

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

Engineering at Meta

DECEMBER 6, 2023

To improve people’s ability to scrutinize us, we also support the Code Verify browser extension for our web-based end-to-end encrypted messaging, to give security researchers greater confidence that the code version that they are assessing is being used globally.

Building 145

Building Designing Consulting Accessibility 145

dbt Developer Hub

APRIL 19, 2023

Don’t repeat yourself** : Dimensions can be easily re-used with other fact tables to avoid duplication of effort and code logic. Step 1: Create model files ​ Let’s create the new dbt model files that will contain our transformation code. We can then build the OBT by running dbt run.

Building 145

Building PostgreSQL BI Database 145

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JULY 23, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. In Hex you can use SQL, Python, R, and no-code visualization together to explore, transform, and model data.

Building 130

Building Machine Learning SQL Python 130

Snowflake

JUNE 5, 2024

Enterprises can utilize gen AI to extract more value from their data and build conversational interfaces for customer and employee applications. It provides access to industry-leading large language models (LLMs), enabling users to easily build and deploy AI-powered applications. Adopting AI solutions can enhance overall efficiency.

Coding 93

Coding Building Management Government 93

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

The Pragmatic Engineer

NOVEMBER 26, 2024

model was released, which has superior code generation compared to ChatGPT. To get a sense of how preferences of developers might have shifted, I asked: “If you're a dev: what is your favorite coding editor with GenAI features (that help your work)? An AI-first code editor, rapidly gaining popularity.

Software Engineer 290

Software Engineer Software Engineering Engineering Coding 290

Start Data Engineering

OCTOBER 22, 2022

Manage cloud infrastructure with code 4. Data project template 2.1. Prerequisites 2.2. Setup infra 2.3. Tear down infra 3. Set up data infrastructure 3.1. Run data infra on your laptop with containers 3.2. Set up development workflow 4.1. CI: Automated tests & checks before the merge with GitHub Actions 4.2. Database migrations 5.

Project 147

Project Data Engineer Data Engineering Building 147

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Other infrastructure:  Primarily AWS ( S3 for cloud object storage, Parameter Store for hierarchical storage, Elastic Container Service ( ECS ) for container deployment and orchestration) The team manages AWS via infrastructure-as-a-code Pulumi. Tech stack.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

The Pragmatic Engineer

SEPTEMBER 26, 2023

Every day, there’s more code at a tech company, not less. This means more repositories are needed, which are fast enough to build and work with, but which increase fragmentation. However, monorepos result in codebases growing large, so that even checking out the code or updating to the head can be time consuming.

Cloud 258

Cloud Software Engineer Software Engineering Cloud Computing 258

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform. Backend code I wrote and pushed to prod took down Amazon.com for several hours. and hand-rolled C -code. At the time, this approach was our best effort to deliver code on the nascent web. 

Engineering 302

Engineering Bytes Cloud Computing AWS 302

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. From learning to code in Australia, to working in Silicon Valley How did I learn to code?

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In August 2021 we open-sourced our code indexing system Glean. Glean collects information about source code and provides it to developer tools through an efficient and flexible query language.

Coding 55

Coding Programming Language SQL Programming 55

Christophe Blefari

MARCH 15, 2024

Building Meta’s GenAI infrastructure — 2x 24k GPU clusters and it's growing. Coding data pipelines is faster than renting connector catalogs — This is something I've always believed. I'm speechless. This is Croissant. It's inspirational.

Metadata 272

Metadata Datasets Data Data Warehouse 272

The Pragmatic Engineer

MAY 1, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineer 200

Software Engineer Software Engineering Programming Language Media 200

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Bun has other contributors, but Jared writes the lion’s share of code.

Programming Language 255

Programming Language Project Coding Engineering 255

Data Engineering Podcast

SEPTEMBER 10, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Datafold : ![Datafold]([link]

BI 208

BI SQL Data Machine Learning 208

Start Data Engineering

JUNE 24, 2024

Features crucial to building and maintaining data pipelines 2.1. Define where to run your code 2.2.3. Introduction 2. Schedulers to run data pipelines at specified frequency 2.2. Orchestrators to define the order of execution of your pipeline tasks 2.2.1. Define the order of execution of pipeline tasks with a DAG 2.2.2.

Data Pipeline 208

Data Pipeline Coding Building Data 208

Analytics Vidhya

APRIL 17, 2023

Introduction Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data science code. It uses best practices of software engineering to build production-ready data science pipelines. This article will give you a glimpse of Kedro framework using news classification tasks.

Data Science 221

Data Science Software Engineer Software Engineering Machine Learning 221

ArcGIS

SEPTEMBER 18, 2024

Learn how to build cross-street data into a geocoding locator using ArcGIS Data Interoperability and its no-code approach.

Building 80

Building Data Coding IT 80

Data Engineering Podcast

OCTOBER 29, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 189

Database BI SQL Machine Learning 189

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Go to dataengineeringpodcast.com/dagster today to get started.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. The services and systems need to be kept up to date, but so does the code that controls their behavior. Search for "Code Commentst" in your podcast player or go to dataengineeringpodcast.com/codecomments today to subscribe.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. This episode is supported by Code Comments, an original podcast from Red Hat. My thanks to the team at Code Comments for their support. Starburst : ![Starburst

Data Governance 147

Data Governance Government Data Lake High Quality Data 147