Lyft Engineering

MARCH 27, 2023

However, at Lyft we have come to realize that load testing in production is a powerful tool to prepare systems for unexpected bursty traffic and peak events. In the context of this article we mean any tool that creates traffic to stress test systems and see how they perform at the limits of their capacity.

Coding 132

Coding Database Systems Engineering 132

Confluent

MAY 13, 2021

Failures are inevitable in any system, and there are various options for mitigating them automatically. This is made possible by event-driven applications leveraging Apache Kafka® and built with fault tolerance […].

Kafka 95

Kafka Systems 95

Pinterest Engineering

AUGUST 8, 2024

In our previous system, which operated on a daily budget allocation model, the system relied on predicting daily budgets for individual users on a daily basis, constraining the flexibility and responsiveness required for dynamic user engagement and content changes. Figure 1 below shows the overview of the system architecture.

Systems 43

Systems Machine Learning Utilities Architecture 43

Confluent

MAY 29, 2024

The dual-write problem can arise in any distributed system. Fortunately, it has solutions in event sourcing & the transactional outbox & listen-to-yourself patterns.

Systems 94

Systems IT Kafka 94

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

Cloudyard

JULY 20, 2023

Read Time: 1 Minute, 38 Second During the previous post we discussed the usage of EVENT table with Python and Javascript procedure. In continuation of the same we will explore how we can leverage the EVENT tables in SQL scripting. We will use the EVENT tables to log the error and other details through SQL scripting procedure.

SQL 52

SQL Python Systems Designing 52

Cloudyard

FEBRUARY 27, 2024

This blog post showcases a novel approach combining Snowflake’s Event Logging and Dynamic Tables to automate the creation and population of dynamic tables based on Copy operations. Event-Driven Logging The journey begins with an innovative approach to log and monitor data copy operations.

Data Warehouse 52

Data Warehouse Utilities Architecture Data Management 52

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system. Building the KPay payment system.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. This rule empowers microservices to change their database schemas however they like without worrying about breaking other systems. What are Events?

Kafka 52

Kafka ETL Tools BI Database 52

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 94

Systems Machine Learning Data Mining Algorithm 94

Engineering at Meta

AUGUST 9, 2023

Explore is one of the largest recommendation systems on Instagram. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and flexible. locally popular media), which further contributes to system scalability.

Systems 92

Systems Media Algorithm Machine Learning 92

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven. Event-first FaaS? Event-first FaaS.

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

FEBRUARY 24, 2022

A few years ago I helped build an event-driven system for gym bookings. The pitch was that we were building a better experience for both the gym members booking different […].

Systems 124

Systems Building Data 124

Uber Engineering

SEPTEMBER 23, 2021

With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more. This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog.

Kafka 135

Kafka Process Systems Engineering 135

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. In the event of server failure, Kafka can autonomously restart itself, ensuring uninterrupted data flow. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Can you describe what you are building at Opaque Systems and the story behind it? Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Scott Logic

APRIL 22, 2024

I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment. I’ll use the term message in this blog as a generic term for asynchronous communication that may be state or commands or events.

AWS 52

AWS Metadata Systems Kafka 52

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. By exploring the benefits of using Data Mesh for your event streaming architecture, this article will help you decide if it’s the right solution for your organization.

Architecture 52

Architecture Generalist Government Datasets 52

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. We need a systemized source of truth that can simulate various performance factors across compute, storage, and network collectively. For instance, the AI Research SuperCluster for AI research.

Systems 105

Systems Architecture Designing Utilities 105

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. Can you describe what Immunai is and the story behind it?

Systems 100

Systems Software Engineering Software Engineer Data Warehouse 100

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Being event first.

Architecture 92

Architecture Programming Kafka Database-centric 92

Data Engineering Podcast

JULY 6, 2020

Summary There are an increasing number of use cases for real time data, and the systems to power them are becoming more mature. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home.

Systems 100

Systems Kafka SQL Data Engineer 100

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 119

Kafka Systems Architecture Building 119

Scott Logic

AUGUST 5, 2024

The rise in popularity of observability has gone hand in hand with the rise of microservices and event-driven architectures. But although it’s a relatively new kid on the block, is observability hyped or a necessary evolution in managing modern software systems? Links from this episode What is Observability? –

Systems 59

Systems Data Collection Architecture Government 59

Cloudera

SEPTEMBER 20, 2021

In Part 1, the discussion is related to: Serial and Parallel Systems Reliability as a concept, Kafka Clusters with and without Co-Located Apache Zookeeper, and Kafka Clusters deployed on VMs. . Serial and Parallel Systems Reliability . Serial Systems Reliability. Serial Systems Reliability.

Kafka 115

Kafka Systems Utilities Bytes 115

The Pragmatic Engineer

OCTOBER 18, 2024

And yet, substitute Apple with Automattic, App Store with WordPress.org and Spotify with one of the most popular WordPress plugins: and Automattic’s CEO is accused of orchestrating events similar to above. This event is shameful and unprecedented in the history of open source on the web. Open source theft? Source: X What next?

Engineering 318

Engineering Government Project AWS 318

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. What is an OS snapshot?

Systems 55

Systems Metadata Datasets Designing 55

Knowledge Hut

JANUARY 18, 2024

CISSP stands for Certified Information Systems Security Professional, and it is a certification in cyber security. This professional certification is developed and offered by (ISC)2, also known as International Information Systems Security Certification Consortium. are interconnected and can be a part of your job description.

Systems 52

Systems Certification Consulting Education 52

Knowledge Hut

APRIL 25, 2024

Intrusion detection systems (IDS) are designed to identify suspicious and malicious activity through network traffic. So, let's get to know the meaning of an intrusion detection system and how it works. What Is a n Intrusion Detection System? Why Are Intrusion Detection Systems Important? and how it works.

Systems 52

Systems Recruitment Machine Learning Certification 52

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 87

Systems Metadata Media Kafka 87

Precisely

MAY 30, 2024

For IBM AIX and IBM i systems, solutions like Assure MIMIX provide seamless cloud transition, real-time data replication, and flexible recovery options to ensure continuous system protection. Are you considering leveraging the cloud alongside your existing IBM AIX and IBM i systems infrastructure?

Systems 69

Systems Cloud Cloud Computing Education 69

Confluent

MARCH 19, 2019

When should you use an event? I will attempt to articulate in layman’s terms what an event-driven architecture (EDA) is and contrast it with service-oriented architecture (SOA). In essence, this is an attempt to differentiate and/or associate APIs with events. Is not life simply a series of events? If it can, should it?

Architecture 110

Architecture Telecommunication Transportation Database 110

U-Next

FEBRUARY 16, 2023

As my thoughts started wandering around our Banking systems and Cosmos Bank Cyber-attack 2018. Also, the recovery also gets affected as there is a lag of almost 24 months between fraud and detection. A robust fraud detection and monitoring system is required. The system should time and again monitor and report audit authorities.

Banking 74

Banking Systems Education Government 74

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. This brings with it a vast number of subtle ways that errors can creep in.

Systems 100

Systems Building Scala Java 100