LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineer 363

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. The changes messed up all major databases in some unexpected way. Still, I’m puzzled by how long the system has been down.

Banking 221

Banking Utilities Database Engineering 221

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. No more shipping and praying, you can now know exactly what will change in your database!

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

The Pragmatic Engineer

OCTOBER 17, 2024

The current database includes 2,000 server types in 130 regions and 340 zones. Results are stored in git and their database, together with benchmarking metadata. Databases:  SQLite files used to publish data Duck DB to query these files in the public APIs Cockroach DB : used to collect and store historical data.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. Andy Pavlo researches autonomous database systems, and out of that research he created OtterTune to find the optimal set of parameters to use for your specific workload.

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Cloudera

OCTOBER 29, 2021

CDP Operational Database (COD) is an autonomous transactional database powered by Apache HBase and Apache Phoenix. COD is easy-to-provision and self-managing, that means developers can provision a new database instance within minutes and start creating prototypes quickly. You can access COD right from your CDP console.

Database 97

Database Amazon Web Services AWS Cloud 97

Data Engineering Podcast

MAY 25, 2021

The problem is that in order for the models to integrate with external systems their internal state has to be translated into a lower dimension. To eliminate this impedance mismatch Edo Liberty founded Pinecone to build database that works natively with vectors. Can you start by describing what Pinecone is and the story behind it?

Database 100

Database Building Data Warehouse Machine Learning 100

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. Start trusting your data with Monte Carlo today!

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

AUGUST 6, 2022

For analytical systems there are decades of investment in data warehouses and various modeling techniques. For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. Who is the target audience for this database?

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

MARCH 5, 2022

Summary When you think about selecting a database engine for your project you typically consider options focused on serving multiple concurrent users. Sometimes what you really need is an embedded database that is blazing fast for single user workloads. Can you describe what DuckDB is and the story behind it?

Database 100

Database Data Lake Java Data Engineering 100

Zalando Engineering

APRIL 22, 2024

But our system is event driven, all requests we process are delivered as events via Nakadi. We know if our system runs within its normal limits that we meet our SLOs. If we would control the ingestion of message requests into our system we would be able to process the task in a timely manner.

Algorithm 75

Algorithm Systems Kafka Designing 75

Data Engineering Podcast

MAY 21, 2023

The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 89

Systems Metadata Media Kafka 89

Maxime Beauchemin

JANUARY 7, 2018

When functions are “pure” — meaning they do not have side-effects — they can be written, tested, reasoned-about and debugged in isolation, without the need to understand external context or history of events surrounding its execution. Results may vary depending on how smart your database optimizer is.

Data Engineering 183

Data Engineering Data Engineer Data Process Process 183

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. So if a user requests a piece of information, the request goes to the Database server. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Precisely

DECEMBER 16, 2024

Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. AIOps presents enormous promise, but many organizations face hurdles in its implementation: Complex ecosystems made of multiple, fragmented systems that lack interoperability.

Systems 52

Systems IT Machine Learning Insurance 52

Data Engineering Weekly

AUGUST 6, 2024

CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. However, managing data consistency across microservices can be challenging.

Data Lake 126

Data Lake Data Warehouse Database Data Architecture 126

Data Engineering Podcast

SEPTEMBER 17, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Building 189

Building SQL BI Python 189

Data Engineering Podcast

OCTOBER 15, 2023

Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data.

Process 182

Process Building SQL BI 182

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. What technical systems are required to generate and collect those interactions?

Building 130

Building IT Metadata MongoDB 130

Start Data Engineering

MARCH 27, 2021

Event driven pipelines Lambda function to trigger spark jobs Setup and run Monitoring and logging Teardown Conclusion Further reading References Event driven pipelines Event driven systems represent a software design pattern where a logic is executed in response to an event.

AWS 100

AWS Cloud Storage Database Cloud 100

Netflix Tech

APRIL 27, 2023

Ruchir Jha , Brian Harrington , Yingwu Zhao TL;DR Streaming alert evaluation scales much better than the traditional approach of polling time-series databases. It allows us to overcome high dimensionality/cardinality limitations of the time-series database. Engineers want their alerting system to be realtime, reliable, and actionable.

Database 120

Database Architecture Consulting Systems 120

Cloudera

FEBRUARY 10, 2022

Apache NiFi’s rich processor library provides Azure focused processors like ADLS Gen2, Event Hub, Blob Storage or Cosmos DB out of the box. Figure 2: Moving application log data from Azure Event Hub to ADLS Gen2 and SIEM systems. Figure 3: Moving data from Azure Event Hub to ADLS Gen2. SIEM Optimization.

Cloud 121

Cloud Kafka AWS Data Ingestion 121

DoorDash Engineering

MARCH 7, 2023

While most engineering tooling at DoorDash is focused on making safe incremental improvements to existing systems, in part by testing in production (learn more about our end-to-end testing strategy ), this is not always the best approach when launching an entirely new business line. Set up a locally running containerized PostgreSQL database.

AWS 119

AWS PostgreSQL Database SQL 119

Snowflake

JUNE 10, 2024

Leveraging Snowflake’s Query History, Event Tables , Alerts and Notifications as the telemetry foundation, Snowflake Trail provides enhanced visibility into data quality, pipelines and applications. Events are all within Snowflake with no need for additional data transfer. Explore logs in your event table easily within Snowsight.

Python 107

Python Java Hadoop Coding 107

Netflix Tech

AUGUST 1, 2022

This article gives an overview of the system. As the system evolves to solve more and more use cases, we have expanded its scope to handle not only the CDC use cases but also more general data movement and processing use cases such that: Events can be sourced from more generic applications (not only databases).

Process 109

Process Transportation Kafka Entertainment 109

Christophe Blefari

AUGUST 24, 2024

Events ✨ As you may know I'm co-organising the Forward Data Conference on November 25th in Paris. Get tickets for Forward Data Conference Transition to another event. Genealogy of databases — like a subway map that depicts all databases changes since Prehistory.

BI 130

BI Food Data SQL 130

Data Engineering Weekly

JULY 14, 2024

Canva: How Canva collects 25 billion events per day Canva writes about its event collection infrastructure capabilities, handling 25 billion events per day (800 billion events per month) with 99.999% uptime. Practical insights into scaling data integration for generative AI with Nexla and Amazon Bedrock.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Data Engineering Podcast

DECEMBER 28, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. No more shipping and praying, you can now know exactly what will change in your database!

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Scott Logic

APRIL 22, 2024

I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment. I’ll use the term message in this blog as a generic term for asynchronous communication that may be state or commands or events.

AWS 52

AWS Metadata Systems Kafka 52

Jesse Anderson

DECEMBER 12, 2022

Google looked over the expanse of the growing internet and realized they’d need scalable systems. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly. We lacked a scalable pub/sub system. In the beginning, there was Google. They created MapReduce and GFS in 2004.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

JUNE 12, 2022

With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. What are the types of storage and data systems that you integrate with?

Unstructured Data 130

Unstructured Data MongoDB MySQL Scala 130

Cloudera

MARCH 15, 2023

Introduction Many modern application designs are event-driven. An event-driven architecture enables minimal coupling, which makes it an optimal choice for modern, large-scale distributed systems. Send an event to notify other services about the new order. These services might be responsible for checking the inventory (eg.

PostgreSQL 96

PostgreSQL Kafka Database Data 96

DoorDash Engineering

MARCH 27, 2024

Real-time event processing is a critical component of a distributed system’s scalability. At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. One of the challenges we face, however, is how to properly validate the system before going live.

Kafka 103

Kafka Architecture Algorithm Systems 103

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great effect, but they are inherently limited in their utility due to their nature as storage systems.

Metadata 130

Metadata MongoDB MySQL Scala 130

LinkedIn Engineering

MAY 9, 2023

Historically, Profile backend had employed a centralized cache using memcached as a read cache between the application and the database. We implemented the following guardrails to protect important components of the integrated system and prevent a cache failure from creating a cascade of failures.

Database 133

Database Project Datasets Designing 133