The Pragmatic Engineer

SEPTEMBER 26, 2023

Every day, there’s more code at a tech company, not less. However, monorepos result in codebases growing large, so that even checking out the code or updating to the head can be time consuming. Concern about code leaks. Open source VS Code Server. In 2021, Microsoft open sourced VS Code Server.

Cloud 258

Cloud Software Engineer Software Engineering Cloud Computing 258

The Pragmatic Engineer

OCTOBER 10, 2023

However, Martin had not written a line of production code for the last four years, as he’s taken on the role of CEO, and heads up observability scaleup Chronosphere – at more than 250 people and growing. From learning to code in Australia, to working in Silicon Valley How did I learn to code?

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

Engineering at Meta

DECEMBER 10, 2024

Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.

Systems 74

Systems Architecture Engineering Data Pipeline 74

Data Engineering Podcast

MARCH 3, 2024

As a listener to the Data Engineering Podcast you can get a special discount off regular priced and late bird tickets by using the promo code dataengpod20. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Promo Code: dataengpod20 Starburst : ![Starburst When is AI the wrong choice?

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Knowledge Hut

MAY 22, 2024

The operating system is loaded via a bootstrapping procedure, often known as booting. An operating system or other larger program, such as a boot loader, is loaded by this application. After loading the operating system into the main memory and turning on the computer, it is prepared to accept user commands.

Systems 52

Systems Process Programming Portfolio 52

The Pragmatic Engineer

NOVEMBER 9, 2023

What would you do if you learned your company is up to something illegal like stealing customer funds, or you’re asked to make code changes that will enable something illegal to happen, like misleading investors, or defrauding customers? Sign up to The Pragmatic Engineer to get articles like this earlier in your inbox.

Software Engineer 227

Software Engineer Software Engineering Engineering Coding 227

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. Guided by the principle that the orchestrator shouldn’t get in your way, Prefect is the only tool of its kind to offer the flexibility to write code as workflows.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

MARCH 10, 2024

As a listener to the Data Engineering Podcast you can get a special discount off regular priced and late bird tickets by using the promo code dataengpod20. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Promo Code: dataengpod20 Dagster : ![Dagster Article: What is Lakehouse Management?:

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

The Pragmatic Engineer

AUGUST 10, 2023

Ayhan visualized this data and observed a definite fall in all metrics: page views, visits, questions asked, votes. Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it. Booking.com says a systems migration is the reason for the delay.

Retail 176

Retail Utilities Software Engineer Software Engineering 176

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. All the code you get with the Dataflow sample workflows is fully functional, adjusted to your environment and isolated from other sample workflows that others generated.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Data Engineering Podcast

DECEMBER 21, 2021

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. What is your overarching design philosophy for the API of the Cube system? What are some of the data modeling steps that are needed in the source systems?

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Netflix Tech

JULY 22, 2024

By Jun He , Natallia Dzenisenka , Praneeth Yenugutala , Yingyi Zhang , and Anjali Norwood TL;DR We are thrilled to announce that the Maestro source code is now open to the public! A workflow definition is defined in a JSON format. Maestro supports dynamic parameters with code injection, which is super useful and powerful.

Machine Learning 104

Machine Learning Coding Data Pipeline Java 104

Engineering at Meta

OCTOBER 17, 2023

Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. So, how did we efficiently and safely remove all of the code and data related to Moments without adversely affecting Meta’s other products and services? different data types stored in 21 different data systems.

Coding 122

Coding Engineering Portfolio Data Schemas 122

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Towards Data Science

AUGUST 8, 2023

Pioneering Data Observability: Data, Code, Infrastructure, & AI The four dimensions of data observability: data, code, infrastructure, and ai? Outlining the past, present, and future of architecting reliable data systems. You look at the code. Image courtesy of the author. No dice, so what’s next?

Coding 70

Coding Data Software Engineer Software Engineering 70

Tweag

MAY 16, 2023

The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.). That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure.

MySQL 134

MySQL Metadata Data Validation Coding 134

Data Engineering Podcast

OCTOBER 8, 2021

Business intelligence tools have provided this capability for years, but they don’t offer a means of exposing those metrics to other systems. Metriql is an open source project that provides a headless BI system where you can define your metrics and share them with all of your other processes.

BI 100

BI Business Intelligence SQL Data Warehouse 100

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Christophe Blefari

MAY 18, 2023

Organisations are systems with inputs and outputs and a formula. Scaling Uber metrics systems (w/ Pinot) — uMetric migration from ES to Pinot. uMetric manages definition, discovery, computation, verification and serving. The analytics goal is to model correctly business. He does a great retro.

Data 130

Data BI Consulting Data Science 130

DataKitchen

APRIL 13, 2021

Because it is such a new category, both overly narrow and overly broad definitions of DataOps abound. Apache Oozie — An open-source workflow scheduler system to manage Apache Hadoop jobs. Reflow — A system for incremental data processing in the cloud. Other Simple Orchestration Tools. Telm.ai — Telm.ai Development Testing Only.

Consulting 124

Consulting Machine Learning Data Science Data Pipeline 124

Tweag

NOVEMBER 1, 2023

The last important remaining piece to explore is the merge system. In this post, we’ll see how to use the Nickel merge system to write reusable configuration modules, and why the merge approach seems to be more adapted for modular configurations than plain functions, despite Nickel being a functional language.

Metadata 113

Metadata Programming Programming Language Coding 113

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests. Kent Beck ?

Coding 68

Coding SQL Software Engineer Software Engineering 68

Knowledge Hut

MAY 26, 2024

Kubernetes is a system for managing and orchestrating containerized applications across a cluster of nodes. The system is designed to simplify life cycle management, allowing developers to focus on their application code rather than infrastructure maintenance. It was designed by Google to manage and schedule containers at scale.

Metadata 52

Metadata Certification Accessible Accessibility 52

The Pragmatic Engineer

APRIL 13, 2023

” My take is that in the way Covid-19 was an unforeseen ‘black swan’ event, so was the boom in tech and in VC-funding in 2021, which was definitely impacted by the pandemic, thanks to businesses and consumers shifting to digital, as a result of the lockdowns making in-person activities difficult and non-practical.

Finance 193

Finance Media Software Engineer Software Engineering 193

Netflix Tech

JUNE 13, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 2 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience. This is where large-scale system migrations come into play.

Systems 114

Systems Utilities Entertainment Coding 114

Knowledge Hut

JANUARY 22, 2024

Let us understand this with the help of example code – class myClass: def instance_method(self): return “Instance method is called”, self The method “instance_method” is a regular instance method. Unlike this variable in C++, self is not a keyword rather it is more of a coding convention.

Python 98

Python Programming Language Programming Coding 98

Data Engineering Weekly

JULY 28, 2024

I like testing people on their practical knowledge rather than artificial coding challenges. The blog Prompt Engineering for a Better SQL Code Generation With LLMs is a pretty good guide on applying prompt engineering to improve productivity. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 111

Data Engineering Data Engineer Engineering Database-centric 111

Christophe Blefari

JANUARY 20, 2024

In order to do it you have to ask Bard to write and execute code in background but to activate the code execution feature "you're at the mercy of the classifier" This is our future, being at the mercy of classifiers. Do I want my warehouse to be able to send emails without my global orchestration system to be aware of it?

Data 130

Data Python Google Cloud Algorithm 130

Knowledge Hut

JUNE 9, 2024

Systems engineering is a diverse field of information technology engineering and engineering management that focuses on the design and administration of composite systems throughout their life cycles. Let's explore what is system engineer? They will manage many teams, testing, and product or system development analysis.

Systems 52

Systems Engineering Telecommunication Electronics 52

Data Engineering Podcast

MAY 22, 2022

In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful. What are the patterns that you and the community have established to encourage discovery and reuse of granular task definitions?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Netflix Tech

OCTOBER 18, 2022

Due to its popularity, the number of workflows managed by the system has grown exponentially. The scheduler on-call has to closely monitor the system during non-business hours. As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits.

Java 85

Java Data Machine Learning Systems 85

Cloudera

NOVEMBER 13, 2020

The commonly-accepted best practice in database system design for years is to use an exhaustive search strategy to consider all the possible variations of specific database operations in a query plan. As with most caching systems, two common problems eventually arise: keeping the cache data up to date, and managing the size of the cache.

Metadata 144

Metadata Coding SQL Database 144

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. This is a great exploration of what it means to treat your data platform as a living system and apply state of the art engineering to it.

BI 100

BI Data Warehouse Metadata SQL 100

Engineering at Meta

OCTOBER 31, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. This process can be unexpectedly difficult, because large software systems are inevitably interconnected.

Data 107

Data Metadata Coding Systems 107

Knowledge Hut

FEBRUARY 6, 2024

As a matter of fact, DevOps way of defining every phase (coding, testing, infrastructure provisioning, deployment, and monitoring) as code also makes it easier to rollback a versioned code in case of disaster recovery and make the environment easily scalable, portable and secure. Who can become a DevOps Engineer?

Engineering 98

Engineering Programming Language Amazon Web Services Google Cloud 98

Data Engineering Podcast

APRIL 12, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values.

Data Warehouse 100

Data Warehouse Data Pipeline BI Data Engineering 100

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130