KDnuggets

NOVEMBER 25, 2024

A data pipeline is just a series of procedures that transport data from one location to another, frequently changing it along the way. Building data pipelines is a very important skill that you should learn as a data engineer.

Data Pipeline 120

Data Pipeline Building Transportation Data 120

Engineering at Meta

AUGUST 5, 2024

We opted for RDMA Over Converged Ethernet version 2 (RoCEv2) as the inter-node communication transport for the majority of our AI capacity. This backend fabric utilizes the RoCEv2 protocol, which encapsulates the RDMA service in UDP packets for transport over the network.

Transportation 131

Transportation Designing Architecture Data Ingestion 131

databricks

JANUARY 14, 2025

Electronic products are evolving at lightning speed, driven by an insatiable demand for new consumer devices, energy, transport, robotics, connectivity, data and beyond.

Electronics 107

Electronics Designing Transportation Manufacturing 107

Lyft Engineering

FEBRUARY 24, 2023

Optimal transport At the core of the Lyft platform is the matching algorithm that dispatches drivers to satisfy rider demand. First, define a non-negative valued transport function γ(i, j) and a cost function c(i, j). That is, we’d like our transport function to move drivers but not riders. 2021) to rectify these shortcomings.

Transportation 101

Transportation Utilities Systems Algorithm 101

KDnuggets

APRIL 11, 2022

Check out the collection of the best data repositories on healthcare, natural language, neuroscience, physics, social network, sports, time series, transportation, miscellaneous, and super data repositories.

Transportation 112

Transportation Healthcare Data Data Science 112

Cloudera

DECEMBER 14, 2021

Roads and Transport Authority, Dubai. The Roads and Transport Authority (RTA) operating in Dubai wanted to apply big data capabilities to transportation and enhance travel efficiency. For this, the RTA transformed its data ingestion and management processes. .

Transportation 111

Transportation Telecommunication Banking Data Lake 111

Netflix Tech

JUNE 7, 2021

The sidecar has been implemented by leveraging the highly performant eBPF along with carefully chosen transport protocols to consume less than 1% of CPU and memory on any instance in our fleet. The choice of transport protocols like GRPC, HTTPS & UDP is runtime dependent on characteristics of the instance placement.

Transportation 131

Transportation AWS Cloud Kafka 131

Engineering at Meta

JANUARY 31, 2023

General high-level architecture Any asynchronous computing platform is composed of the following building blocks: Ingestion and storage Transport and routing Computation Ingestion and storage Our platform is responsible for accepting the workloads and storing them for execution. The transport layer consisted of a component called “Dispatcher.”

Transportation 107

Transportation Architecture Data Warehouse MySQL 107

Knowledge Hut

DECEMBER 21, 2023

IoT: Overview IoT has numerous applications in various sectors such as healthcare, agriculture, transportation, manufacturing, and smart cities. Some of the popular smart city projects include smart transportation, smart energy, and smart waste management. If you want to know more about IoT, check out online IoT training.

Coding 98

Coding Transportation Medical Retail 98

Confluent

AUGUST 26, 2021

When companies need help with their vehicle fleets—including transport, storage, or renewing expired registrations—they don’t want to have to deal with multiple vehicle logistics providers. For these companies, ACERTUS provides […].

Transportation 105

Transportation Architecture 105

Snowflake

JANUARY 12, 2024

Cargill Ocean Transportation (OT) manages 650 ships at sea every single day. To support IMO’s mission, the maritime industry adopted a new metric—Carbon Intensity Indicator (CII)—which measures a vessel’s efficiency when it is transporting cargo and passengers.

Transportation 121

Transportation Manufacturing Government Accessible 121

Snowflake

OCTOBER 24, 2023

Founded nearly 70 years ago, Werner Enterprises is a North American transportation and logistics leader that operates a fleet of almost 8,300 trucks and 30,000 trailers out of 16 terminals across the United States. The company generates a massive amount of data on the constantly changing, real-time location of each of its assets.

Transportation 115

Transportation Data Warehouse Python Cloud 115

Netflix Tech

AUGUST 1, 2022

It reads events from the upstream transports and applies some business logic to each of them. An intermediate processor writes data to another transport. Processors A processor is a Flink Job. It contains a reusable unit of data processing logic.

Process 109

Process Transportation Kafka Entertainment 109

Snowflake

DECEMBER 18, 2023

Here are 50 Data Leaders to watch in 2024: Amin Venjara , CDO, ADP Andrew Bonnici , Vice President – Product, Platforms, and Data, IGT PlayDigital Andrew Curry , CDO, ExxonMobil Aravind Jagannathan , VP and CDO, Freddie Mac Arno Huhn , Managing Director Data and AI, Schwarz Group Ashok Chennuru , Global Chief Data & Insights Officer, Elevance (..)

Transportation 109

Transportation Insurance Data Architecture Technology 109

Cloudera

AUGUST 11, 2022

To drive these data use cases, the Department of Defense (DoD) communities and branches require a reliable, scalable data transport mechanism to deliver data (from any source) from origination through all points of consumption; at the edge, on-premise, and in the cloud in a simple, secure, universal, and scalable way. edge processing.

Transportation 94

Transportation Data Ingestion Architecture Data 94

Data Engineering Podcast

DECEMBER 28, 2020

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Satori Data Governance Data Masking TLS == Transport Layer Security The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support (..)

Data Governance 130

Data Governance Government Transportation Data 130

Knowledge Hut

APRIL 25, 2024

Benefits of Big Data in Transportation Industry 5: Transportation The benefit of big data is more to the transportation industry. Increased Knowledge Big data and IoT have the potential to inform customers or users about the most efficient mode of transportation at any given time. Why Choose Big Data?

Big Data 97

Big Data Entertainment Transportation Banking 97

Confluent

MARCH 26, 2019

The transport and database components are pluggable. The transport component is used to ingest tracing data. Transport protocol options include an HTTP API, a Kafka producer and others. Zipkin Architecture Overview. Both Apache Cassandra and Elasticsearch can be used as data stores for the trace data.

Kafka 111

Kafka Transportation Metadata Consulting 111

Towards Data Science

FEBRUARY 9, 2024

The Platform Integration Data Store Transformation Orchestration Presentation Transportation Observability Closing What’s changed? The future of the transportation layer seems destined to intersect with APIs, creating a scenario where API endpoints generated via SQL queries become as common as exporting .csv

Building 98

Building Transportation Data Lake Metadata 98

Snowflake

DECEMBER 18, 2023

Here are 50 Data Leaders to watch in 2024: Amin Venjara , CDO, ADP Andrew Bonnici , Vice President – Product, Platforms, and Data, IGT PlayDigital Andrew Curry , CDO, ExxonMobil Aravind Jagannathan , VP and CDO, Freddie Mac Arno Huhn , Managing Director Data and AI, Schwarz Group Ashok Chennuru , Global Chief Data & Insights Officer, Elevance (..)

Transportation 98

Transportation Insurance Data Architecture Technology 98

The Pragmatic Engineer

OCTOBER 24, 2023

It's like asking a tricycle maker to pilot a spaceship because, well, it's all transportation. Tesla engineers thus resorted to measuring what most experts will tell you is completely meaningless: the quantity of code contributed to Twitter by each engineer.

Coding 236

Coding Software Engineer Software Engineering Media 236

Cloudera

JULY 24, 2024

People need to get to work, go to the doctor, and get groceries, and it’s up to their local transportation department to ensure they make it to their destinations reliably. Similarly, transportation agencies reduce downtime through innovations like automatic inflating tire systems; so it’s about time we innovate too.

Transportation 79

Transportation Metadata Cloud Systems 79

Picnic Engineering

FEBRUARY 6, 2024

Our goods are transported in boxes with standardised sizes, called totes. Within this 250 ms, a couple of things need to happen in order: Send information on the tote to the Transport System. Raise a divert decision request to the Warehouse Control System by the Transport System. Execute the divert decision.

Cloud 97

Cloud Transportation Cloud Computing Systems 97

RandomTrees

DECEMBER 17, 2024

Risk factors in manufacturing, storage, and transportation activities can be easily monitored through GenAI-powered IoT sensors and systems. Personalized Solutions: GenAI is able to assess past data and propose a custom quality assurance plan for each product. Food Safety Management- through efficient real-time monitoring.

Food 52

Food Manufacturing Machine Learning Algorithm 52

Lyft Engineering

JUNE 8, 2023

Lyft was founded in 2012 and went public in 2019, with the mission to improve people’s lives with the world’s best transportation. We’re looking for driven engineers to fortify our European operations and solve some of the hardest problems in building large distributed systems to support rideshare, mapping, and more.

Transportation 104

Transportation Algorithm Machine Learning Engineering 104

U-Next

MARCH 12, 2023

An administration uses XML to label data, SOAP to transport messages, and WSDL to show how accessible particular administrations are. Features of Web Services There are several characteristics of web services: XML-Based Records are transported and represented using XML in web services.

Cloud Computing 98

Cloud Computing Architecture Cloud Transportation 98

Cloudera

FEBRUARY 3, 2021

Capacity on planes and with ground transportation has been booked, and, of course, this was all done in the absence of the vaccine itself. The COVID-19 vaccine supply chain may go down in history as the largest single undertaking of manufacturing, transport, and logistics. But that’s not the whole story.

Manufacturing 107

Manufacturing Transportation Pharmaceutical Data Consolidation 107

Netflix Tech

APRIL 20, 2020

To achieve that, we are efficiently using ABR (adaptive bitrate streaming) for a better playback experience, DRM (Digital Right Management) to protect our service and TLS (Transport Layer Security) to protect customer privacy and to create a safer streaming experience. is the latest version of the Transport Layer Security protocol.

Electronics 98

Electronics Transportation Media Algorithm 98

Engineering at Meta

FEBRUARY 7, 2023

Transportation buildings, such as airport terminals, parking garages, and metro stations, have a bluish tint to correspond with our new iconography layer. We’re now also color-coding buildings in our “areas of interest” layer, which highlights features in high traffic areas with a creamy orange tint.

Entertainment 137

Entertainment Transportation Data Schemas AWS 137

Engineering at Meta

NOVEMBER 15, 2023

Scaling RoCE Networks for AI Training Adi Gangidi, Production Network Engineer Adi Gangidi provides an overview of Meta’s RDMA deployment based on RoCEV2 transport for supporting our production AI training infrastructure.

Building 91

Building Engineering Transportation Software Engineer 91

Zalando Engineering

MARCH 6, 2025

Each component incorporates end-to-end non-blocking I/O, leveraging Nettys EventLoop with Linux-native Epoll transport. DynamoDB ensures high availability and fast lookups when cache misses occur.

Algorithm 63

Algorithm Architecture Transportation Data Ingestion 63

Snowflake

MARCH 7, 2023

A Snowflake customer in the transportation industry Scania uses data mesh and Snowflake’s Data Cloud to drive transport sustainability. Let’s get to work We only have one liveable planet and we are all in this together.

Transportation 100

Transportation Data Data Lake Retail 100

Engineering at Meta

JANUARY 27, 2023

The talk also covers the connection of our submarine networks to our terrestrial backbone and describes how Meta designs and builds the hierarchies of the optical transport layer built on top of those fiber paths.

Engineering 130

Engineering Software Engineer Software Engineering Transportation 130

Data Engineering Weekly

OCTOBER 11, 2024

The Cloud-Native Modern Data Stack Era: The Rise of Supply Chains and Specialized Services [Supply Chain of Nutella - [link] ] Globalization and advancements in transportation revolutionized manufacturing in the latter half of the 20th century.

Manufacturing 110

Manufacturing Transportation Data Warehouse Unstructured Data 110

Snowflake

SEPTEMBER 19, 2024

The Texas Department of Transportation sought a secure data platform that would provide a single point of truth and help drive actionable analytics and enhance decision-making. In the government and education sector, moving to a modernized data strategy doesn’t have to be a cost-prohibitive, labor-intensive undertaking.

Telecommunication 98

Telecommunication Hospitality Retail Entertainment 98

Engineering at Meta

APRIL 17, 2023

For example, we have been able to characterize the nature of bursts across a number of services in order to understand the intensity of incast and tune transport performance accordingly.

Bytes 115

Bytes Transportation Data Collection Coding 115

Knowledge Hut

JANUARY 28, 2024

It offers the best results to companies operating in diverse sectors, including healthcare, energy, transportation, financial services, and more. The solution was to prioritize the reopening of Highway 5, the principal corridor for transportation of goods and people, by creating scopes, work breakdown structures, and schedules for each site3.

Project 98

Project Management Hospitality Consulting 98