Engineering at Meta

APRIL 17, 2023

How it works: Millisampler comprises userspace code to schedule runs, store data, and serve data, and an eBPF-based tc filter that runs in the kernel to collect fine-timescale data. The user code attaches the tc filter and enables data collection. Millisampler collects a variety of metrics.

Bytes 116

Bytes Transportation Data Collection Coding 116

Engineering at Meta

SEPTEMBER 12, 2023

Meta’s Native Assurance team regularly performs manual code reviews as part of our ongoing commitment to improve the security posture of Meta’s products. For example, it uses SELinux policies to reduce the attack surfaces exposed to unprivileged code running on the device.

Bytes 134

Bytes Coding Programming Process 134

Rock the JVM

DECEMBER 17, 2023

The UDP header is fixed at 8 bytes and contains a source port, destination port, the checksum used to verify packet integrity by the receiving device, and the length of the packet which equates to the sum of the payload and header. flip () println ( s "[server] I've received ${content.limit()} bytes " + s "from ${clientAddress.toString()}!

Scala 90

Scala Bytes Java Coding 90

Tweag

JANUARY 18, 2023

reading and writing to a byte stream). Say we have some arbitrary Python class, a first approximation may be to define encode and decode methods: from typing import Self class MyClass : def encode ( self ) - > bytes : # Implementation goes here. classmethod def decode ( cls , data : bytes ) - > Self : # Implementation goes here.

Python 102

Python Bytes Software Engineer Software Engineering 102

LinkedIn Engineering

JUNE 15, 2023

An Avro file is formatted with the following bytes: Figure 1: Avro file and data block byte layout The Avro file consists of four “magic” bytes, file metadata (including a schema, which all objects in this file must conform to), a 16-byte file-specific sync marker, and a sequence of data blocks separated by the file’s sync marker.

Datasets 102

Datasets Bytes Process Data Ingestion 102

Cloudera

MARCH 2, 2022

Impala has always focused on efficiency and speed, being written in C++ and effectively using techniques such as runtime code generation and multithreading. For instance, in both the struct s above the largest member is a pointer of size 8 bytes. Total size of the Bucket is 16 bytes. Folding data into pointers.

Data Warehouse 89

Data Warehouse Bytes Data Business Intelligence 89

Engineering at Meta

JANUARY 26, 2023

Chart 2: Bytes logged per second via Legacy versus Tulip. We can see that while the number of logging schemas remained roughly the same (or saw some organic growth), the bytes logged saw a significant decrease due to the change in serialization format. Reader (like logger) comes in two flavors , (a) code generated and (b) generic.

Bytes 107

Bytes Data Engineering Coding 107

Pinterest Engineering

NOVEMBER 22, 2023

See the graph below, which shows the compaction read and write bytes on a cluster when it is bootstrapping for the first time. Figure 10: compaction read and write bytes showing non zero values as soon as host starts up. We decided to do the former because the code change was very minimal and the path to production would be fast.

Database 111

Database Bytes Kafka Architecture 111

Snowflake

JUNE 20, 2024

With familiar DataFrame-style programming and custom code execution, Snowpark lets teams process their data in Snowflake using Python and other programming languages by automatically handling scaling and performance tuning. It provided us insights as to code compatibility and allowed us to better estimate our migration time.”

Data Engineering 111

Data Engineering Data Engineer Scala Engineering 111

Zalando Engineering

DECEMBER 20, 2015

To help our thousands of non-tech employees better understand our many tech topics and terms, one of our Hack Week teams created a training program called Let’s Talk About Code, Baby. Zalando’s many tech-driven projects and initiatives — from logistics to customer care to marketing analysis — are cross-departmental.

Coding 52

Coding Bytes SQL Programming 52

Netflix Tech

MARCH 6, 2019

That is, all mounted files that were opened and every single byte range read that MezzFS received. Finally, MezzFS will record various statistics about the mount, including: total bytes downloaded, total bytes read, total time spent reading, etc. Actions ?—?MezzFS The different colors mean different things?—?green

Media 87

Media Bytes Process Accessibility 87

Pinterest Engineering

FEBRUARY 23, 2023

These advancements fit well with Pinterest use cases  —  enabling faster connection establishment (time to first byte of first request), improved congestion control (large media as we have), multiplexing without TCP head-of-line blocking (multiple downloads at the same time), and continued in-flight requests when pinners’ device network/ip changes.

Bytes 141

Bytes Media Software Engineering Software Engineer 141

Confluent

MAY 29, 2019

We’ll demonstrate using Gradle to execute and test our KSQL streaming code, as well as building and deploying our KSQL applications in a continuous fashion. The first requirement to tackle: how to express dependencies between KSQL queries that exist in script files in a source code repository. Managing KSQL dependencies.

Kafka 96

Kafka Management Bytes SQL 96

Workfall

SEPTEMBER 26, 2023

Fast to code : It enables considerable speed increases in development. Short : It reduces code duplication. Robust : It delivers production-ready code with interactive documentation. UVicorn: UVicorn is a fast ASGI server for running Python web applications, designed for high performance and compatibility with asynchronous code.

Python 85

Python Bytes Coding Project 85

DoorDash Engineering

AUGUST 15, 2023

The Cilium project has an exceptional cilium/ebpf Golang library that compiles and interacts with eBPF probes within Golang code. It has easy-to-use go:generate integration that runs Clang to compile eBPF C code into the BTF format, subsequently wrapping BTF artifacts in an easy-to-use Go package to load the probes.

Bytes 84

Bytes PostgreSQL Coding Database 84

Snowflake

FEBRUARY 27, 2024

For example, you can specify billing events in your app code and charge consumers based entirely on their usage of your application. This API-based billing allows providers to charge and meter based on nearly any value their app offers. This is a flexible, usage-based pricing model to allow providers to price-to-value their applications.

Bytes 122

Bytes Electronics Banking Data 122

Rock the JVM

JULY 26, 2023

HMAC-based One Time Password (HOTP) The H in HOTP stands for HMAC (Hash-based Message Authentication Code). We can now use hotp to generate the code. val counter = 5 val code = hotp. generate ( counter ) In the code above, we initialized the counter to 5 and used that to generate the hotp code. verify ( code ) 3.

Scala 92

Scala Java Bytes Algorithm 92

DoorDash Engineering

JANUARY 16, 2024

The sidecar container operates as a plug-and-play solution, seamlessly intercepting, controlling, and transforming all HTTP1/HTTP2/gRPC traffic flowing in and out of DoorDash microservices — all without requiring any modifications to the application code. It’s challenging to build a comprehensive understanding/view of all cross-AZ traffic.

Bytes 84

Bytes Cloud Management PostgreSQL 84

Confluent

OCTOBER 24, 2019

They offer several functions that are included in Topology and can be used in different applications, such as getting a Java UUID from the fixed bytes ID in the schema. Rust is a compiled language, so code needs to be compiled first in order to be executed. Using Java interop, these classes make working with the data in Clojure easier.

Kafka 18

Kafka Java Banking Bytes 18

Lyft Engineering

MARCH 11, 2024

Instead, we chose to use an envoy circuitbreaker , which returns an HTTP 503 code immediately to the downstream caller. However, this would add to our downtime during cutover, as it would take some time for our service to scale back up and be ready to serve traffic. The diff_bytes is 0 now!

Bytes 81

Bytes PostgreSQL AWS Database 81

Towards Data Science

APRIL 11, 2023

Job information after creating a clustered table(Image by author) Execution details after creating a clustered table(Image by author) From the job info and execution details, you can see the number of bytes processed and records scanned reduced significantly(from 1.5 GB to 55 MB and 7M to 260k).

Bytes 91

Bytes Google Cloud Datasets Programming Language 91

Netflix Tech

SEPTEMBER 3, 2021

If a consumer is only interested in production titles and format, they can set a FieldMask with paths “title” and “format”: [link] Masking fields Please note, even though code samples in this blog post are written in Java, demonstrated concepts apply to any other language supported by protocol buffers. Our protobuf message definition (.proto

Designing 98

Designing Java Bytes Utilities 98

Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. The changes are being streamed via Logical Replication to a blue client and a pink client respectively.

PostgreSQL 139

PostgreSQL Java Database Bytes 139

Towards Data Science

JANUARY 26, 2023

The code block below includes snippets for: creating a file with random data and uploading it to Amazon S3 (using boto3), iterating over all of the samples sequentially, and sampling the data at non-sequential file offsets. The code block below demonstrates the use of S5cmd with the concurrency set to 10. client('s3') s3.upload_file('2GB.bin',

Cloud Storage 98

Cloud Storage Big Data Cloud AWS 98

Cloudera

AUGUST 14, 2020

This framework does not require any code changes to the system-under-test that is being validated. Over time we can do more intrusive whitebox testing by enabling and disabling various join points and delay-points within the Ozone code. No changes to Ozone code required for simulating failures. How does it work?

Hadoop 96

Hadoop Bytes Metadata Programming Language 96

Netflix Tech

FEBRUARY 25, 2021

The output of an encoder is a sequence of bytes, called an elementary stream, which can only be parsed with some understanding of the elementary stream syntax. Each rectangle represents a ‘brand’ (indicated by a four character code in bold), and its required set of tools (indicated by a ‘+’ line). Figure 1?—?Simplified Figure 2?—?Illustrating

Technology 76

Technology Bytes Media Entertainment 76

Cloudera

NOVEMBER 15, 2021

The International Data Corporation (IDC) estimates that by 2025 the sum of all data in the world will be in the order of 175 Zettabytes (one Zettabyte is 10^21 bytes). Seagate Technology forecasts that enterprise data will double from approximately 1 to 2 Petabytes (one Petabyte is 10^15 bytes) between 2020 and 2022.

Unstructured Data 90

Unstructured Data Pipeline-centric Database-centric Entertainment 90

Rock the JVM

FEBRUARY 22, 2023

So, all the code snippets in this article will use the following logger: static final Logger logger = LoggerFactory. So, all the code snippets in this article will use the following logger: static final Logger logger = LoggerFactory. Moreover, we’ll also use Lombok to reduce the boilerplate code when dealing with checked exceptions.

Java 145

Java Programming Coding Scala 145

LinkedIn Engineering

APRIL 18, 2023

li-minion : Installable python agent which gets installed on all 300K + hosts, It is wrapped & packaged as RPM with customized code which automatically discovers relevant master hosts and generates minion config on every start. This product enforces security checks on clients modules to ensure clients are following safe coding practices.

MySQL 103

MySQL Python Bytes Kafka 103

Confluent

JUNE 13, 2019

Much of the code used by modern astronomers is written in Python, so the ZTF alert distribution system endpoints need to at least support Python. We built our alert distribution code in Python, based around Confluent’s Python client for Apache Kafka. Another requirement for the alert distribution system is ease of use for the end user.

Kafka 102

Kafka Bytes Python Data Pipeline 102

Cloudera

OCTOBER 6, 2020

All code is in my github repo. Going through The Code. Hope you find it useful, Happy coding!! *For development ease of use, you can also use the Phoenix query server instead of COD. The query server is a small build of phoenix that is meant for development purposes only, and data is deleted in each build. . Simple as that.

Database 81

Database Building Bytes NoSQL 81

Ripple Engineering

AUGUST 19, 2021

The original implementation of the non-leaf nodes had a C++ class that looked a bit like this: struct SHAMapInnerNode { std::bitset<16> isBranch_; SHAMapHash hashes_[16]; std::shared_ptr<SHAMapTreeNode> cachedChildren_[16]; } To traverse the tree, the code looks at the 256-bit key four bits at a time.

Bytes 52

Bytes Coding Designing Manufacturing 52

Confluent

SEPTEMBER 27, 2019

The best place is inside the event streaming platform itself, so that we don’t have to audit each client to make sure their application code has respected all the rules. However, these schemas are only enforced as “agreement” between the clients and are totally agnostic to brokers, which still see all messages as entirely untyped byte arrays.

Kafka 16

Kafka Data Governance Bytes Government 16

Grouparoo

MAY 13, 2021

However, the the most popular text encoding ( Latin1 or utf8 ) on the most popular MySQL database engine ( innodb ) assumed that 3 bytes was enough to store every character 2 , and once utf8mb4 came along with characters like ? 4 bytes were needed to store each character. Specifically, 767/4 = 191 characters!

Database 52

Database Bytes MySQL Database-centric 52

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. We provide the functions: prefix to reference the subproject directory with our code. jar Zip file size: 5849 bytes, number of entries: 5.

Kafka 89

Kafka Java Bytes SQL 89

Zalando Engineering

JUNE 1, 2016

Ensure a fast Time to First Byte: Request fragments in parallel and stream them as soon as possible, without blocking the rest of the page. How the layout service works In order to achieve a fast Time to First Byte, the service has to asynchronously fetch multiple fragments, assemble their response streams, and output the final output stream.

Bytes 52

Bytes Scala Coding Systems 52

ProjectPro

JANUARY 24, 2023

Numeric data consists of four sub-types: Integer type (INT64) Numeric type (NUMERIC DECIMAL) Bignumeric type (BIGNUMERIC BIGDECIMAL) Floating point type (FLOAT64) BYTES Although they work with raw bytes rather than Unicode characters, BYTES also represent variable-length data. Deploy the model and monitor its performance.

Bytes 52

Bytes Google Cloud Data Warehouse Cloud Storage 52