Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. Meta recently announced they have made Buck2 open-source.

Systems 141

Systems Building Java Programming Language 141

Knowledge Hut

JANUARY 11, 2024

In today's digital age, the rise of the internet has given us two powerful mediums for expressing creativity: web design & graphic design. Web design involves creating websites, interfaces, & online applications while graphic design entails creating visual designs for print & digital media.

Designing 96

Designing Digital Media Media Programming Language 96

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. Subscribe here. and hand-rolled C -code.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Can you start by outlining what elements of observability are most relevant for dbt projects? Over the past ~3 years there were numerous data observability systems/products created. How is Elementary designed/implemented?

Project 130

Project Data Lake High Quality Data Data Workflow 130

The Pragmatic Engineer

AUGUST 1, 2023

Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. We put a lot of emphasis on communication and prioritization and the ability to unblock yourself or your team – this comes on top of the programming and design skills.

Software Engineer 245

Software Engineer Software Engineering Engineering Designing 245

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 95

Systems Media Machine Learning Data Warehouse 95

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. Arcadia gives Meta’s researchers and engineers valuable insights into the performance of AI models and workloads in an AI cluster – enabling data-driven decision making in the design of AI clusters.

Systems 112

Systems Architecture Designing Utilities 112

Start Data Engineering

MAY 28, 2024

Project demo 3. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Introduction 2. Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3.

Data Pipeline 219

Data Pipeline Python Building Data 219

Data Engineering Podcast

OCTOBER 23, 2022

Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. How do you balance the need to provide value with the long-term impacts of incomplete or underinformed modeling decisions made in haste at the beginning of a project?

Project 130

Project Metadata MongoDB MySQL 130

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.

Project 98

Project Management Programming Designing 98

The Pragmatic Engineer

MARCH 12, 2024

The title of the book takes aim at the “myth” that software development can be measured in “man months,” which Brooks disproves in the pages that follow: “Cost [of the software project] does indeed vary as the product of the number of men and the number of months. Progress does not. Two secretaries.

Engineering 274

Engineering Programming Language Hospitality Programming 274

Knowledge Hut

OCTOBER 29, 2023

Choosing the best computer science project topic is critical to the success of any computer science student or employee. After all, the more engaging and interesting topic, the more likely it is that students or employees will be able to stay motivated and focused throughout the duration of the project.

Computer Science 111

Computer Science Coding Project Hospitality 111

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Can you describe what you are building at Opaque Systems and the story behind it? When is Opaque the wrong choice?

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. If you've learned something or tried out a project from the show then tell us about it! Trusted by teams of all sizes, including Comcast and Doordash.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? If you've learned something or tried out a project from the show then tell us about it!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 176

Recruitment Java Coding Project 176

The Pragmatic Engineer

AUGUST 22, 2023

Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

The Pragmatic Engineer

SEPTEMBER 22, 2023

It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects. Bun has no such constraint. If you are one of these people: thank you!

Programming Language 255

Programming Language Project Coding Engineering 255

Knowledge Hut

JANUARY 11, 2024

For most of us, the role of a Project Manager is quite well defined. But how many of us know the role a project manager plays in an Agile project? Some other questions that often boggle budding Agilists are, exactly how different a product owner is different from a project manager? And are these roles interchangeable?

Project 98

Project Management Certification Process 98

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. How have the design and goals/scope changed since you first started working on it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

The Pragmatic Engineer

OCTOBER 17, 2024

  This grant is designed to “support entrepreneurs, tech-geeks, developers, and socially engaged people, who are capable of challenging the way we search and discover information and resources on the internet” The team is tiny; only three people. Internal comms: Chat: Slack Coordination / project management: Linear 3.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. When a cluster's usage expands beyond the original designed capacity, what are the options/procedures for expanding that capacity? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. Want to see Starburst in action?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Weekly

SEPTEMBER 26, 2024

The collective design choices and ideas lead to a comprehensive overview of thinking about designing data infrastructure with a privacy-first approach. Most cloud providers offer built-in encryption options and key management systems (KMS) , making it easier to stay compliant without sacrificing security. sensitivity, project).

Systems 69

Systems Engineering Data Warehouse Architecture 69

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

The Pragmatic Engineer

APRIL 9, 2024

a YouTuber doing design videos has designers watching, or a software engineer like me writing on software engineering has developers reading) These "creators" would vet people looking for jobs And they would also vet companies looking to hire. For all the companies who trusted me to help with your hiring: thank you!

Software Engineer 271

Software Engineer Software Engineering Engineering Recruitment 271

Data Engineering Podcast

JUNE 9, 2024

To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. How have the requirements of generative AI shifted the demand for streaming data systems? What role does Flink play in the architecture of generative AI systems?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 21, 2024

How have the design and goals of the product changed since you started it? How do you manage the personalization of the AI functionality in your system for each user/team? If you've learned something or tried out a project from the show then tell us about it! Can you describe how you have architected the Shortwave platform?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. number/scale of systems, types of data, AI) Data governance can often become an exercise in boiling the ocean. Trusted by teams of all sizes, including Comcast and Doordash.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Engineering at Meta

OCTOBER 15, 2024

At the Open Compute Project (OCP) Global Summit 2024, we’re showcasing our latest open AI hardware designs with the OCP community. These innovations include a new AI platform, cutting-edge open rack designs, and advanced network fabrics and components. For example, Llama 3.1

Deep Learning 131

Deep Learning Designing Architecture Systems 131

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. What were the design goals and constraints that led you to this architecture? If you've learned something or tried out a project from the show then tell us about it!

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

SEPTEMBER 3, 2023

The dlt project was created to eliminate overhead and bring data integration into your full control as a library component of your overall data system. The obvious comparison is with systems like Singer/Meltano/Airbyte in the open source space, or Fivetran/Matillion/etc. in the commercial space. in the commercial space.

Data Integration 162

Data Integration BI SQL Python 162

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. I was still recovering from this failure in 2015, when a friend who designed the first version of Uber– Jelle Prins – pinged me, and said Uber was kickstarting an engineering office in Amsterdam.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253