Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. Meta recently announced they have made Buck2 open-source.

Systems 141

Systems Building Java Programming Language 141

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. Subscribe here. and hand-rolled C -code.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Powered by Trino, the query engine Apache Iceberg was designed for, Starburst is an open platform with support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Start Data Engineering

MAY 28, 2024

Use DuckDB to process data, not for multiple users to access data 4.2. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Building efficient data pipelines with DuckDB 4.1. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB?

Data Pipeline 219

Data Pipeline Python Building Data 219

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 95

Systems Media Machine Learning Data Warehouse 95

Striim

SEPTEMBER 11, 2024

A data pipeline is a systematic sequence of components designed to automate the extraction, organization, transfer, transformation, and processing of data from one or more sources to a designated destination. Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

DoorDash Engineering

FEBRUARY 22, 2023

Additionally, the Fulfillment engineering team recently released this article about how they use the CockroachDB to scale their system. We also got similar benefits by using the CockroachDB in our system. This system would ensure that we could provide an accurate and fresh view of all sellable items in stores to our customers.

Designing 125

Designing Database Architecture SQL 125

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

FEBRUARY 18, 2024

What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? What are the differences in terms of pipeline design/access and usage patterns when using a Trino/Iceberg lakehouse as compared to other popular warehouse/lakehouse structures?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

The Pragmatic Engineer

MARCH 12, 2024

Tools and approaches at our disposal, which didn’t exist in 1975, or were not widespread in 1995, include: Git – the now-dominant version control system used by much of the industry, with exceptions for projects with very large assets, like video games Code reviews : these became common in parallel with version control.

Engineering 274

Engineering Programming Language Hospitality Programming 274

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 94

Systems Machine Learning Data Mining Algorithm 94

Snowflake

MAY 16, 2024

Connecting government agencies through the power of integrated technologies By design, government and education programs are regulated and complex. Together, we drive success for our government partners, local and national businesses, and hundreds of thousands of families who can receive access to important academic support.

Education 83

Education Accessible Accessibility Recruitment 83

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. When a cluster's usage expands beyond the original designed capacity, what are the options/procedures for expanding that capacity? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Weekly

SEPTEMBER 26, 2024

The collective design choices and ideas lead to a comprehensive overview of thinking about designing data infrastructure with a privacy-first approach. Privacy and access management within data infrastructure is not just a best practice; it's a necessity. Why care about privacy? Let's examine them in more detail.

Systems 69

Systems Engineering Data Warehouse Architecture 69

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. How have the design and goals/scope changed since you first started working on it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

The Pragmatic Engineer

MARCH 19, 2024

Today, full subscribers got access to a comprehensive Senior-and-above tech compensation research. Source: Cognition So far, all we have is video demos, and accounts of those with access to this tool. In 1959, the programming language COBOL was designed by software engineer Grace Hopper. And COBOL was just one of many attempts.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. OpenSSL : the cryptography and SSL/TLS toolkit comes with a built-in performance benchmarking capability Lmbench : tools for performance analysis for UNIX/POSIX system.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. Want to see Starburst in action?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 176

Recruitment Java Coding Project 176

Engineering at Meta

MARCH 12, 2024

We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We have been openly designing our GPU hardware platforms beginning with our Big Sur platform in 2015.

Building 145

Building Portfolio Utilities Data Storage 145

The Pragmatic Engineer

APRIL 9, 2024

On top of the ability to post jobs, they had access to The Pragmatic Engineer Talent Collective. Companies stayed customers longer, as they were interested in having access to the "latest drops" in the talent collective. (A There was a similar trend in companies paying to access the talent collective and job board.

Software Engineer 271

Software Engineer Software Engineering Engineering Recruitment 271

Data Engineering Podcast

JUNE 9, 2024

To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. How have the requirements of generative AI shifted the demand for streaming data systems? What role does Flink play in the architecture of generative AI systems?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Snowflake

APRIL 24, 2024

Members of the Snowflake AI Research team pioneered systems such as ZeRO and DeepSpeed , PagedAttention / vLLM , and LLM360 which significantly reduced the cost of LLM training and inference, and open sourced them to make LLMs more accessible and cost-effective for the community. license provides ungated access to weights and code.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Engineering at Meta

JUNE 27, 2023

Sapling: Scaling version control Sapling is a version control system that can scale to huge sizes, but also emphasizes usability. There are three main components to Sapling – a server, a client, and a virtual file system. The final component is the virtual file system.

Java 132

Java Programming Language Algorithm Coding 132

Data Engineering Podcast

APRIL 21, 2024

How have the design and goals of the product changed since you started it? How do you manage the personalization of the AI functionality in your system for each user/team? How have the design and goals of the product changed since you started it? Can you describe how you have architected the Shortwave platform?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Engineering at Meta

OCTOBER 15, 2024

At the Open Compute Project (OCP) Global Summit 2024, we’re showcasing our latest open AI hardware designs with the OCP community. These innovations include a new AI platform, cutting-edge open rack designs, and advanced network fabrics and components. By sharing our designs, we hope to inspire collaboration and foster innovation.

Deep Learning 131

Deep Learning Designing Architecture Systems 131

The Pragmatic Engineer

MARCH 13, 2023

The issue is that banks (not only SVB) aren’t designed to work like this: they cannot fulfil simultaneous requests for withdrawals by all (or even most) customers. The bank’s systems start to be overloaded to the point of customers not being able to log on and transfer. It's an all out bank run.

Banking 192

Banking Insurance Portfolio Media 192

Engineering at Meta

DECEMBER 6, 2023

Our aim is to ensure that everyone’s personal messages on Messenger can only be accessed by the sender and the intended recipients, and that everyone can be sure the messages they receive are from an authentic sender. End-to-end encryption isn’t about the technology at its core.

Building 145

Building Designing Consulting Accessible 145

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. number/scale of systems, types of data, AI) Data governance can often become an exercise in boiling the ocean. Trusted by teams of all sizes, including Comcast and Doordash.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Robinhood

OCTOBER 10, 2024

When Robinhood was founded, we set out to build a platform that gives everyone access to the financial markets. Throughout the design process, we channeled the bold and disruptive spirit that defines Robinhood to create something completely different that elevates our brand and challenges the industry as a whole.

Banking 123

Banking Retail Designing Finance 123

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. These formats are transforming how organizations manage large datasets.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. What were the design goals and constraints that led you to this architecture? What were the design goals and constraints that led you to this architecture?

Database 162

Database Technology Data Lake High Quality Data 162

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 101

Datasets Computer Science Systems Kafka 101

Data Engineering Podcast

MAY 12, 2024

The services and systems need to be kept up to date, but so does the code that controls their behavior. Powered by Trino, the query engine Apache Iceberg was designed for, Starburst is an open platform with support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Management 147

Management Data Lake High Quality Data Machine Learning 147