Netflix Tech

SEPTEMBER 18, 2024

The first level is a hashed string ID (the primary key), and the second level is a sorted map of a key-value pair of bytes. A future blog post will describe the chunking architecture in more detail, including its intricacies and optimization strategies. The idempotency token ties all these writes together into one atomic operation.

Bytes 104

Bytes Metadata Database Data 104

WeCloudData

MARCH 1, 2025

quintillion bytes of data are generated every day. The world is becoming increasingly dependent on data, about 2.5 Data is shaping our decisions, from personalized shopping experiences to checking weather forecasts before leaving home. All of these data science applications have a life cycle to follow.

Data Science 52

Data Science Project Bytes Data 52

Pinterest Engineering

FEBRUARY 23, 2023

These advancements fit well with Pinterest use cases  —  enabling faster connection establishment (time to first byte of first request), improved congestion control (large media as we have), multiplexing without TCP head-of-line blocking (multiple downloads at the same time), and continued in-flight requests when pinners’ device network/ip changes.

Bytes 141

Bytes Media Software Engineering Software Engineer 141

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. So which static site generator to choose?

Engineering 40

Engineering Bytes AWS Python 40

Netflix Tech

SEPTEMBER 24, 2021

Our previous tech blog Packaging award-winning shows with award-winning technology detailed our packaging technology deployed on the streaming side. Writable MezzFS As described in a previous blog post, MezzFS is a tool developed by Netflix that allows cloud storage objects to be mounted as local files via FUSE.

Cloud 96

Cloud Bytes Cloud Storage Media 96

Zalando Engineering

NOVEMBER 8, 2023

Introduction This blog post describes a recent contribution from Zalando to the Postgres JDBC driver to address a long-standing issue with the driver’s integration with Postgres’ logical replication that resulted in runaway Write-Ahead Log (WAL) growth. However as you may imagine, this blog post concerns a path that is anything but happy.

PostgreSQL 139

PostgreSQL Java Database Bytes 139

Cloudera

JANUARY 17, 2024

In this blog we will dive into how CDF-PC’s support for NiFi reporting tasks can be used to monitor key metrics in Prometheus and Grafana. By using component_name and “Hello World Prometheus,” we’re monitoring the bytes received aggregated by the entire process group and therefore the flow. Select the nifi_amount_bytes_received metric.

Bytes 106

Bytes Architecture Building Designing 106

Workfall

SEPTEMBER 26, 2023

Reading Time: 9 minutes In this blog, we will cover: What are Server-Sent Events? We’re taking in 16 bytes of data at a time from the stream. This function will provide basic units of data in the form of raw bytes. These bytes can then be converted into a readable JSON format. appeared first on The Workfall Blog.

Python 85

Python Bytes Coding Project 85

Confluent

MAY 29, 2019

If you want to follow along and execute all the commands included in this blog post (and the next), you can check out this GitHub repository , which also includes the necessary Docker Compose functionality for running a compatible KSQL and Confluent Platform environment using the recently released Confluent 5.2.1. Sample repository.

Kafka 96

Kafka Management Bytes SQL 96

DoorDash Engineering

JANUARY 16, 2024

In this blog post, we describe the journey DoorDash took using a service mesh to realize data transfer cost savings without sacrificing service quality. This led us to use a number of observability tools, including VPC flow logs , ebpf agent metrics , and Envoy networking bytes metrics to rectify the situation.

Bytes 84

Bytes Cloud Management PostgreSQL 84

DoorDash Engineering

AUGUST 15, 2023

For a more detailed introduction to BPF portability and CO-RE, see Andrii Nakryiko’s blog post on the subject. We also have an unmarshalling function to convert the raw bytes from the kernel into our structure. The post BPFAgent: eBPF for Monitoring at DoorDash appeared first on DoorDash Engineering Blog.

Bytes 84

Bytes PostgreSQL Coding Database 84

Lyft Engineering

MARCH 11, 2024

This blog would be of immense help to understand what happens under the hood with AWS blue/green deployment! The diff_bytes is 0 now! We now need to reset sequences in, which we accomplished with the following script: [link] This ensures that the sequence starts from the last entry of the individual tables.

Bytes 81

Bytes PostgreSQL AWS Database 81

Cloudera

NOVEMBER 15, 2021

This blog discusses quantifications, types, and implications of data. The International Data Corporation (IDC) estimates that by 2025 the sum of all data in the world will be in the order of 175 Zettabytes (one Zettabyte is 10^21 bytes). The post The Rise of Unstructured Data appeared first on Cloudera Blog.

Unstructured Data 90

Unstructured Data Pipeline-centric Database-centric Entertainment 90

Cloudera

MAY 9, 2022

Refer to the YARN – The Capacity Scheduler blog to understand these configuration settings.) . This can be tuned using the user limit factor of the YARN queue (refer the details in Capacity Scheduler blog ). Tez determines the reducers automatically based on the data (number of bytes) to be processed.

Bytes 118

Bytes SQL Professional Services Utilities 118

Snowflake

APRIL 18, 2024

This blog post explores how Snowflake can help with this challenge. In the cloud, computing can be measured in various ways, like bytes scanned or CPU cycles. But what if security teams didn’t have to make tradeoffs? Detection and investigation processing: Security teams depend on detection rules to find important events automatically.

Data Lake 109

Data Lake Data Ingestion Bytes Cloud Computing 109

Netflix Tech

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. We may go into detail on this subject in one of our future blog posts. The next section describes how this is achieved.

Bytes 95

Bytes Datasets Metadata Data 95

Confluent

OCTOBER 24, 2019

I’ve written an event sourcing bank simulation in Clojure (a lisp build for Java virtual machines or JVMs) called open-bank-mark , which you are welcome to read about in my previous blog post explaining the story behind this open source example. Make sure it is indeed an ID and that the Value matches the expected type Fixed , with 16 bytes.

Kafka 18

Kafka Java Banking Bytes 18

Cloudera

OCTOBER 6, 2020

In this blog, I will demonstrate how COD can easily be used as a backend system to store data and images for a simple web application. The post Building a Simple CRUD web application and image store using Cloudera Operational Database and Flask appeared first on Cloudera Blog.

Database 81

Database Building Bytes NoSQL 81

Data Engineering Weekly

DECEMBER 3, 2023

Github writes an excellent blog to capture the current state of the LLM integration architecture. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications. I experienced similar drawbacks to what Lyft is talking about in Druid. Rebalancing, the awkward middle child.

Data Engineering 71

Data Engineering Data Engineer Engineering Bytes 71

Netflix Tech

FEBRUARY 25, 2021

By Cyril Concolato Introduction In previous blog posts, our colleagues at Netflix have explained how 4K video streams are optimized , how even legacy video streams are improved and more recently how new audio codecs can provide better aural experiences to our members. Figure 1?—?Simplified

Technology 76

Technology Bytes Media Entertainment 76

LinkedIn Engineering

SEPTEMBER 7, 2023

Espresso System Overview Figure 1 is a high-level overview of the Espresso ecosystem, which includes the online operation section of Espresso (the main focus of this blog post). Improvements to Encode/Decode performance This section focuses on the performance improvements we made when converting bytes to Http objects and vice versa.

Bytes 88

Bytes Transportation Utilities Java 88

Pinterest Engineering

NOVEMBER 28, 2023

Pyoung = Seden / Ralloc where Pyoung is the period between young GC, Seden is the size of Eden and Ralloc is the rate of memory allocations (bytes per second). To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site.

Kafka 106

Kafka Bytes Architecture Software Engineer 106

Netflix Tech

SEPTEMBER 3, 2021

If a consumer is only interested in production titles and format, they can set a FieldMask with paths “title” and “format”: [link] Masking fields Please note, even though code samples in this blog post are written in Java, demonstrated concepts apply to any other language supported by protocol buffers. Field names are not included.

Designing 98

Designing Java Bytes Utilities 98

Cloudera

SEPTEMBER 20, 2021

In this blog series, we will discuss each of these deployments and the deployment choices made along with how they impact reliability. The post Apache Kafka Deployments and Systems Reliability – Part 1 appeared first on Cloudera Blog. There are many ways that Apache Kafka has been deployed in the field.

Kafka 116

Kafka Systems Utilities Bytes 116

Netflix Tech

MAY 5, 2021

We will cover the different namespaces of Netflix Drive in more detail in a subsequent blog post. Data Store Characteristics Netflix Drive relies on a data store that allows streaming bytes into files/objects persisted on the storage media. The transfer mechanism for transport of bytes is a function of the data store.

Metadata 72

Metadata Bytes Media Cloud Storage 72

Scott Logic

APRIL 5, 2024

Introduction This series of blog posts discusses the methods of estimating carbon emissions of end-user devices. After intending to write a single blog post, the research journey prompted me to reconsider how to present this to an audience. js is a javascript library that returns an estimated CO2e value for a web page.

Bytes 52

Bytes Systems Designing Data Storage 52

Towards Data Science

APRIL 30, 2024

This blog post is my note after reading the paper: The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. In the rest of this blog, we will see how Google enables this contribution. See you next blog!

Google Cloud 92

Google Cloud Process Cloud Lambda Architecture 92

ProjectPro

JANUARY 24, 2023

This blog is your comprehensive guide to Google BigQuery, its architecture, and a beginner-friendly tutorial on how to use Google BigQuery for your data warehousing activities. This blog presents a detailed overview of Google BigQuery and its architecture. Due to this, combining and contrasting the STRING and BYTE types is impossible.

Bytes 52

Bytes Google Cloud Data Warehouse Cloud Storage 52

Data Engineering Weekly

FEBRUARY 5, 2023

The ML for large-scale production systems highlights the improvement made from the existing heuristic in the YouTube cache replacement algorithm with a new hybrid algorithm that combines a simple heuristic with a learned model, improving the byte miss ratio at the peak by ~9%. The blog talks about four types of architecture.

Data Engineering 58

Data Engineering Data Engineer Engineering Food 58

Cloudera

AUGUST 14, 2020

The target could be a particular Node (network endpoint), a file-system, a directory, a data-file or a byte-offset range within a given data-file. The post Apache Ozone Fault Injection Framework appeared first on Cloudera Blog. A Typical flow control for Apache Ozone using this Fault Injection Framework looks like this: .

Hadoop 96

Hadoop Bytes Metadata Programming Language 96

Towards Data Science

JANUARY 26, 2023

Check out this informative blog for more details on how S5cmd works and its significant performance advantages. Here we show how to download specific byte-ranges of the file using the Boto3 get_object data streaming API. CPU cores and TCP connections). The S5cmd concurrency flag allows for controlling the download speed.

Cloud Storage 98

Cloud Storage Big Data Cloud AWS 98

Cloudera

OCTOBER 6, 2020

This is a guest blog post, authored by John Zantey, Director and Co-founder, Qabsu. Globally, there are quintillions of bytes of data being generated and collected, every day. The post Collaboration is Key to Reducing Pain and Finding Value in Data appeared first on Cloudera Blog. collect, enrich, report, serve, and predict).

Bytes 67

Bytes Education Cloud Data 67

Confluent

JULY 10, 2019

The repository’s README contains a bit more detail, but in a nutshell, we check out the repo and then use Gradle to initiate docker-compose : git clone [link] cd kafka-examples git checkout confluent-blog./gradlew jar Zip file size: 5849 bytes, number of entries: 5. jar Zip file size: 11405084 bytes, number of entries: 7422.

Kafka 89

Kafka Java Bytes SQL 89

Cloudera

APRIL 23, 2021

sent 11,286 bytes received 172 bytes 2,546.22 However, we can continue without enabling TLS for the purpose of this blog. The post HDFS Data Encryption at Rest on Cloudera Data Platform appeared first on Cloudera Blog. [root@ccycloud-4 ~]# rsync -zav --exclude.ssl /var/lib/keytrustee/.keytrustee keytrustee ccycloud-3.cdpvcb.root.hwx.site:/var/lib/keytrustee/.

MySQL 73

MySQL Java Bytes Data 73

Datakin

SEPTEMBER 2, 2021

Blog Data Quality + Data Lineage = Written by Peter Hicks on Sep 2, 2021 In a prior life, I dwelled in the day-to-day cycles of an e-commerce platform. In previous blog posts, we’ve talked before about the importance of understanding data lineage from debugging to privacy and governance.

Bytes 52

Bytes Food Datasets Data Pipeline 52

Cloudera

FEBRUARY 14, 2024

This blog discusses a few problems that you might encounter with Iceberg tables and offers strategies on how to optimize them in each of those scenarios. In a later blog, we will go into details about how to take advantage of the time travel feature. The rest of the blog will go into this in more detail. Opening files is costly.

Bytes 59

Bytes Metadata Data Lake SQL 59

Tweag

APRIL 19, 2023

In this blog post, I will explain the underlying technical challenges and share the solution that we helped implement at kaiko.ai , a MedTech startup in Amsterdam that is building a Data Platform to support AI research in hospitals. A solution is to read the bytes that we need when we need them directly from Blob Storage.

Medical 62

Medical Process Cloud Bytes 62