Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems. ETL workflows), as well as downstream (e.g.

Systems 93

Systems Media Machine Learning Data Warehouse 93

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. This brings with it a vast number of subtle ways that errors can creep in.

Systems 100

Systems Building Scala Java 100

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. These two distinct approaches offer unique strategies for securing buildings and valuables. What are Fail-Safe Door Locks?

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. In this episode Gleb Mezhanskiy shares some valuable advice and insights into how you can build reliable and well-tested data assets with dbt and data-diff.

Project 130

Project Building Data Lake Machine Learning 130

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, or aren't fast enough for interactive use cases.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Monte Carlo

AUGUST 17, 2022

Over the years, I’ve helped companies of all sizes build and maintain data systems—from my days as a data engineer at Facebook to my current role as an end-to-end data solutions consultant. It’s never too early to start building reliability into your data system. The stakes also get higher.

Systems 52

Systems Building Data Data Engineering 52

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. What technical systems are required to generate and collect those interactions?

Building 130

Building IT Metadata MongoDB 130

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. In the event of server failure, Kafka can autonomously restart itself, ensuring uninterrupted data flow.

Kafka 75

Kafka Architecture AWS Transportation 75

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. We use Snowflake as our data warehouse where we build dashboards both for internal use and for customers. So our solution was to start using an intentional contract: Events.

Kafka 52

Kafka ETL Tools BI Database 52

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven. Event-first FaaS? What is FaaS?

Kafka 109

Kafka AWS Architecture Cloud 109

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. In this article, we will explore the advantages and limitations of data mesh, while also providing best practices for building and optimizing a data mesh with Striim.

Architecture 52

Architecture Generalist Government Datasets 52

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Engineering at Meta

AUGUST 9, 2023

Explore is one of the largest recommendation systems on Instagram. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and flexible. locally popular media), which further contributes to system scalability.

Systems 92

Systems Media Algorithm Machine Learning 92

Pinterest Engineering

JANUARY 4, 2024

In 2020, anticipating the growing needs of the business and to simplify our storage offerings, we decided to consolidate our different key-value systems in the company into a single unified service called KVStore. In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator.

Database 93

Database Building Datasets Relational Database 93

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. We need a systemized source of truth that can simulate various performance factors across compute, storage, and network collectively. For instance, the AI Research SuperCluster for AI research.

Systems 105

Systems Architecture Designing Utilities 105

Data Engineering Podcast

JULY 6, 2020

Summary There are an increasing number of use cases for real time data, and the systems to power them are becoming more mature. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Kafka SQL Data Engineering 100

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 119

Kafka Systems Architecture Building 119

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Data evolution.

Architecture 92

Architecture Programming Kafka Database-centric 92

Rockset

SEPTEMBER 2, 2022

I recently had the good fortune to host a small-group discussion on personalization and recommendation systems with two technical experts with years of experience at FAANG and other web-scale companies. Nikhil Garg is CEO and co-founder of Fennel AI , a startup working on building the future of real-time machine learning infrastructure.

Systems 52

Systems Building Machine Learning NoSQL 52

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 78

Building Transportation Data Lake Metadata 78

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Precisely

DECEMBER 28, 2023

This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. That’s where streaming data pipelines come into play.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. What is an OS snapshot?

Systems 55

Systems Metadata Datasets Designing 55

Data Engineering Podcast

FEBRUARY 1, 2021

As the systems that we build continue to grow in scale and complexity the need for reliable and manageable monitoring platforms increases proportionately. In this episode Rob Skillington, CTO of Chronosphere, shares his experiences building metrics systems that provide observability to companies that are operating at extreme scale.

Systems 100

Systems Cloud BI Data Warehouse 100

Knowledge Hut

JANUARY 18, 2024

CISSP stands for Certified Information Systems Security Professional, and it is a certification in cyber security. This professional certification is developed and offered by (ISC)2, also known as International Information Systems Security Certification Consortium. are interconnected and can be a part of your job description.

Systems 52

Systems Certification Consulting Education 52

Snowflake

MAY 20, 2024

This event will use the Homeless Data Integration System (HDIS), which consolidates homelessness data to offer a holistic view of statewide efforts. Not all theater and “executive invite only” sessions will be recorded or made available after the event. Register today and be a part of the data revolution!

Building 85

Building Certification Cloud BI 85

Data Engineering Podcast

JULY 20, 2020

Summary Wind energy is an important component of an ecologically friendly power system, but there are a number of variables that can affect the overall efficiency of the turbines. Michael Tegtmeier founded Turbit Systems to help operators of wind farms identify and correct problems that contribute to suboptimal power outputs.

Systems 100

Systems Machine Learning Manufacturing Algorithm 100

Netflix Tech

MARCH 14, 2023

We build creator tooling to enable these colleagues to focus their time and energy on creativity. They may scrub for frames, shots, or scenes of specific characters, locations, objects, events (e.g. We implemented a batch processing system for users to submit their requests and wait for the system to generate the output.

Media 118

Media Building Algorithm Machine Learning 118

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 87

Systems Metadata Media Kafka 87

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108