Engineering at Meta

AUGUST 5, 2024

We opted for RDMA Over Converged Ethernet version 2 (RoCEv2) as the inter-node communication transport for the majority of our AI capacity. This backend fabric utilizes the RoCEv2 protocol, which encapsulates the RDMA service in UDP packets for transport over the network.

Transportation 132

Transportation Designing Architecture Data Ingestion 132

Uber Engineering

DECEMBER 19, 2019

Every day around the world, millions of trips take place across the Uber network, giving users more reliable transportation through ridesharing, bikes, and scooters, drivers and truckers additional opportunities to earn, employees and employers more convenient business travel, and hungry … The post Uber Infrastructure in 2019: Improving Reliability, (..)

Transportation 127

Transportation Engineering Architecture 127

Engineering at Meta

JANUARY 31, 2023

General high-level architecture Any asynchronous computing platform is composed of the following building blocks: Ingestion and storage Transport and routing Computation Ingestion and storage Our platform is responsible for accepting the workloads and storing them for execution. The transport layer consisted of a component called “Dispatcher.”

Transportation 119

Transportation Architecture Data Warehouse MySQL 119

Cloudera

DECEMBER 14, 2021

Roads and Transport Authority, Dubai. The Roads and Transport Authority (RTA) operating in Dubai wanted to apply big data capabilities to transportation and enhance travel efficiency. For this, the RTA transformed its data ingestion and management processes. .

Transportation 117

Transportation Telecommunication Banking Data Lake 117

Lyft Engineering

FEBRUARY 24, 2023

Optimal transport At the core of the Lyft platform is the matching algorithm that dispatches drivers to satisfy rider demand. First, define a non-negative valued transport function γ(i, j) and a cost function c(i, j). That is, we’d like our transport function to move drivers but not riders. 2021) to rectify these shortcomings.

Transportation 100

Transportation Utilities Systems Algorithm 100

KDnuggets

APRIL 11, 2022

Check out the collection of the best data repositories on healthcare, natural language, neuroscience, physics, social network, sports, time series, transportation, miscellaneous, and super data repositories.

Transportation 116

Transportation Healthcare Data Data Science 116

Netflix Tech

JUNE 7, 2021

The sidecar has been implemented by leveraging the highly performant eBPF along with carefully chosen transport protocols to consume less than 1% of CPU and memory on any instance in our fleet. The choice of transport protocols like GRPC, HTTPS & UDP is runtime dependent on characteristics of the instance placement.

Transportation 131

Transportation AWS Cloud Kafka 131

Knowledge Hut

DECEMBER 21, 2023

IoT: Overview IoT has numerous applications in various sectors such as healthcare, agriculture, transportation, manufacturing, and smart cities. Some of the popular smart city projects include smart transportation, smart energy, and smart waste management. If you want to know more about IoT, check out online IoT training.

Coding 98

Coding Transportation Medical Retail 98

Confluent

AUGUST 26, 2021

When companies need help with their vehicle fleets—including transport, storage, or renewing expired registrations—they don’t want to have to deal with multiple vehicle logistics providers. For these companies, ACERTUS provides […].

Transportation 105

Transportation Architecture 105

Cloudera

AUGUST 11, 2022

To drive these data use cases, the Department of Defense (DoD) communities and branches require a reliable, scalable data transport mechanism to deliver data (from any source) from origination through all points of consumption; at the edge, on-premise, and in the cloud in a simple, secure, universal, and scalable way. edge processing.

Transportation 100

Transportation Data Ingestion Architecture Data 100

Netflix Tech

AUGUST 1, 2022

It reads events from the upstream transports and applies some business logic to each of them. An intermediate processor writes data to another transport. Processors A processor is a Flink Job. It contains a reusable unit of data processing logic.

Process 109

Process Transportation Kafka Entertainment 109

Knowledge Hut

APRIL 25, 2024

Benefits of Big Data in Transportation Industry 5: Transportation The benefit of big data is more to the transportation industry. Increased Knowledge Big data and IoT have the potential to inform customers or users about the most efficient mode of transportation at any given time. Why Choose Big Data?

Big Data 97

Big Data Entertainment Transportation Banking 97

Snowflake

JANUARY 12, 2024

Cargill Ocean Transportation (OT) manages 650 ships at sea every single day. To support IMO’s mission, the maritime industry adopted a new metric—Carbon Intensity Indicator (CII)—which measures a vessel’s efficiency when it is transporting cargo and passengers.

Transportation 99

Transportation Manufacturing Government Accessible 99

Picnic Engineering

FEBRUARY 6, 2024

Our goods are transported in boxes with standardised sizes, called totes. Within this 250 ms, a couple of things need to happen in order: Send information on the tote to the Transport System. Raise a divert decision request to the Warehouse Control System by the Transport System. Execute the divert decision.

Cloud 97

Cloud Transportation Cloud Computing Systems 97

Data Engineering Podcast

DECEMBER 28, 2020

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Satori Data Governance Data Masking TLS == Transport Layer Security The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support (..)

Data Governance 130

Data Governance Government Transportation Data 130

Cloudera

JULY 24, 2024

People need to get to work, go to the doctor, and get groceries, and it’s up to their local transportation department to ensure they make it to their destinations reliably. Similarly, transportation agencies reduce downtime through innovations like automatic inflating tire systems; so it’s about time we innovate too.

Transportation 84

Transportation Metadata Cloud Systems 84

Confluent

MARCH 26, 2019

The transport and database components are pluggable. The transport component is used to ingest tracing data. Transport protocol options include an HTTP API, a Kafka producer and others. Zipkin Architecture Overview. Both Apache Cassandra and Elasticsearch can be used as data stores for the trace data.

Kafka 111

Kafka Transportation Metadata Consulting 111

Engineering at Meta

NOVEMBER 15, 2023

Scaling RoCE Networks for AI Training Adi Gangidi, Production Network Engineer Adi Gangidi provides an overview of Meta’s RDMA deployment based on RoCEV2 transport for supporting our production AI training infrastructure.

Building 105

Building Engineering Transportation Software Engineer 105

Snowflake

OCTOBER 24, 2023

Founded nearly 70 years ago, Werner Enterprises is a North American transportation and logistics leader that operates a fleet of almost 8,300 trucks and 30,000 trailers out of 16 terminals across the United States. The company generates a massive amount of data on the constantly changing, real-time location of each of its assets.

Transportation 98

Transportation Data Warehouse Python Cloud 98

Engineering at Meta

JANUARY 27, 2023

The talk also covers the connection of our submarine networks to our terrestrial backbone and describes how Meta designs and builds the hierarchies of the optical transport layer built on top of those fiber paths.

Engineering 143

Engineering Software Engineer Software Engineering Transportation 143

Cloudera

FEBRUARY 3, 2021

Capacity on planes and with ground transportation has been booked, and, of course, this was all done in the absence of the vaccine itself. The COVID-19 vaccine supply chain may go down in history as the largest single undertaking of manufacturing, transport, and logistics. But that’s not the whole story.

Manufacturing 110

Manufacturing Transportation Pharmaceutical Data Consolidation 110

Engineering at Meta

FEBRUARY 7, 2023

Transportation buildings, such as airport terminals, parking garages, and metro stations, have a bluish tint to correspond with our new iconography layer. We’re now also color-coding buildings in our “areas of interest” layer, which highlights features in high traffic areas with a creamy orange tint.

Entertainment 143

Entertainment Transportation Data Schemas AWS 143

U-Next

MARCH 12, 2023

An administration uses XML to label data, SOAP to transport messages, and WSDL to show how accessible particular administrations are. Features of Web Services There are several characteristics of web services: XML-Based Records are transported and represented using XML in web services.

Cloud Computing 98

Cloud Computing Architecture Cloud Transportation 98

Netflix Tech

APRIL 20, 2020

To achieve that, we are efficiently using ABR (adaptive bitrate streaming) for a better playback experience, DRM (Digital Right Management) to protect our service and TLS (Transport Layer Security) to protect customer privacy and to create a safer streaming experience. is the latest version of the Transport Layer Security protocol.

Electronics 98

Electronics Transportation Media Algorithm 98

Engineering at Meta

APRIL 17, 2023

For example, we have been able to characterize the nature of bursts across a number of services in order to understand the intensity of incast and tune transport performance accordingly.

Bytes 121

Bytes Transportation Data Collection Coding 121

Cloudera

OCTOBER 18, 2021

Roads and Transport Authority, Dubai . Here are our 2021 Data Impact Award finalists challenging the status quo and pushing the boundaries of possibility: . Data for Enterprise AI . Commonwealth Bank of Australia . Internal Revenue Service. Read more about the Data for Enterprise AI category here . Data for Good . Keck Medicine of USC .

Banking 116

Banking Transportation Government Consulting 116

Towards Data Science

FEBRUARY 9, 2024

The Platform Integration Data Store Transformation Orchestration Presentation Transportation Observability Closing What’s changed? The future of the transportation layer seems destined to intersect with APIs, creating a scenario where API endpoints generated via SQL queries become as common as exporting .csv

Building 76

Building Transportation Data Lake Metadata 76

Cloudera

DECEMBER 1, 2023

million), the Louisiana Department of Motor Vehicles (6 million), and Oregon’s Department of Transportation (3.5 This summer’s MOVEit data breach has impacted over 2,600 organizations and 70,500,000 people worldwide, including Maximus (11 million), the state of Maine (1.3 million), among others.

Building 115

Building Government Transportation Data Governance 115

Data Engineering Weekly

OCTOBER 11, 2024

The Cloud-Native Modern Data Stack Era: The Rise of Supply Chains and Specialized Services [Supply Chain of Nutella - [link] ] Globalization and advancements in transportation revolutionized manufacturing in the latter half of the 20th century.

Manufacturing 111

Manufacturing Transportation Data Warehouse Unstructured Data 111

Knowledge Hut

JANUARY 3, 2024

Human randomness and AI computation present a difficulty for robotic transportation. They may be able to relieve us of the strain of physically taxing or repetitive chores as well as advance healthcare, increase the effectiveness of transportation, and allow us more freedom to engage in creative activities.

Entertainment 98

Entertainment Manufacturing Food Transportation 98

Cloudera

OCTOBER 25, 2023

Many of these emissions are driven by industrial and transportation systems reliant on fossil fuels. Sustainable Cities: Cities are responsible for 70 percent of global greenhouse gas emissions. That means that the climate crisis will be won or lost in our urban environments.

Food 108

Food Machine Learning Transportation Manufacturing 108

Netflix Tech

OCTOBER 19, 2020

This was the most important question we considered when building our infrastructure because data sampling policy dictates the amount of traces that are recorded, transported, and stored. Our trace data collection agent transports traces to Mantis job cluster via the Mantis Publish library.

Building 95

Building Transportation Java Metadata 95

Picnic Engineering

MARCH 14, 2023

Most of the automation consists of conveyor belts, lifts and shuttles that transport stock from place to place in the warehouse. It cannot not, however, orchestrate the transport of crates over large distances. For this purpose, a transport layer exists on top of the control layer. The warehouse consists of four layers.

Python 59

Python Transportation Systems Algorithm 59

Knowledge Hut

MAY 31, 2023

AI finds its use in a wide range of applications like marketing , automation, transport, supply chain, and communication, to name a few. Impact: AI can positively impact society, from healthcare and education to finance and transportation. You can make a meaningful contribution by working on AI-based projects.

Project 96

Project Healthcare Deep Learning Transportation 96

The Pragmatic Engineer

OCTOBER 24, 2023

It's like asking a tricycle maker to pilot a spaceship because, well, it's all transportation. Tesla engineers thus resorted to measuring what most experts will tell you is completely meaningless: the quantity of code contributed to Twitter by each engineer.

Coding 186

Coding Software Engineer Software Engineering Media 186

Knowledge Hut

JULY 6, 2023

It is imperative to eliminate money-consuming waste, such as unnecessary stationery, transportation costs, etc., which maximizes savings and profits. Enhanced Customer Experience One of the key ideologies of applying a lean approach to project management is to enhance communication and work from the customer’s perspective.

Project 98

Project Management Manufacturing Transportation 98

Cloudera

JANUARY 21, 2022

Services in each zone use a combination of kerberos and transport layer security (TLS) to authenticate connections and APIs calls between the respective host roles, this allows authorization policies to be enforced and audit events to be captured. These are then supported by the data tier where configuration and key material is maintained.

Architecture 117

Architecture Transportation Certification Government 117