Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Engineering at Meta

DECEMBER 10, 2024

Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.

Systems 74

Systems Architecture Engineering Data Pipeline 74

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 94

Systems Machine Learning Data Mining Algorithm 94

Engineering at Meta

JUNE 19, 2024

We’re introducing parameter vulnerability factor (PVF) , a novel metric for understanding and measuring AI systems’ vulnerability against silent data corruptions (SDCs) in model parameters. But the growing complexity and diversity of AI hardware systems also brings an increased risk of hardware faults such as bit flips.

Systems 97

Systems Deep Learning Manufacturing Architecture 97

Data Engineering Podcast

FEBRUARY 18, 2024

To start, can you share your definition of what constitutes a "Data Lakehouse"? What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? To start, can you share your definition of what constitutes a "Data Lakehouse"?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

MARCH 5, 2024

The storage system is using Capacitor, a proprietary columnar storage format by Google for semi-structured data and the file system underneath is Colossus, the distributed file system by Google. BigQuery separates storage and compute with Google’s Jupiter network in-between to utilize 1 Petabit/sec of total bisection bandwidth.

Bytes 67

Bytes Google Cloud Cloud Storage Utilities 67

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. What are the different places in a data system that schema definitions need to be established? What are the different places in a data system that schema definitions need to be established?

Systems 100

Systems Metadata Building MongoDB 100

Cloudyard

SEPTEMBER 11, 2024

After successfully migrating several views, the data engineering team noticed discrepancy between the Oracle view definitions and their Snowflake counterparts. The team needed a way to: Extract the view definitions from Snowflake. These discrepancies potentially lead to errors or unexpected results when querying views in Snowflake.

Data Warehouse 52

Data Warehouse Data Engineering Data Engineer Engineering 52

The Pragmatic Engineer

APRIL 19, 2023

ChatGPT can be an unexpectedly useful to break down jargon terms: just don't forget to verify its definition, as ChatGPT can make things up. I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager. Otherwise, understand the jargon in simple terms, yourself.

Software Engineer 295

Software Engineer Software Engineering Engineering Media 295

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Monte Carlo

FEBRUARY 23, 2024

And this renewed focus on data quality is bringing much needed visibility into the health of technical systems. GenAI will definitely make us more productive. Over the past several years, data observability tooling has emerged as a must-have for every industry, from retail to security. The opportunities are just boundless.”

Systems 64

Systems Building Retail Data Lake 64

Data Engineering Podcast

MARCH 3, 2024

What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" What are some of the useful clarifying/scoping questions to address when deciding the path to deployment for different definitions of "AI"? "AI" has grown to be an even more overloaded term than it already was.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

The Pragmatic Engineer

AUGUST 10, 2023

Ayhan visualized this data and observed a definite fall in all metrics: page views, visits, questions asked, votes. Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it. Booking.com says a systems migration is the reason for the delay.

Retail 176

Retail Utilities Software Engineer Software Engineering 176

Netflix Tech

JULY 22, 2024

A workflow definition is defined in a JSON format. Maestro combines user-supplied fields with those managed by Maestro to form a flexible and powerful orchestration definition. A Maestro workflow definition comprises two main sections: properties and versioned workflow including its metadata.

Machine Learning 104

Machine Learning Coding Data Pipeline Java 104

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager.

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

Data Engineering Podcast

DECEMBER 21, 2021

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. What is your overarching design philosophy for the API of the Cube system? What are some of the data modeling steps that are needed in the source systems?

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Data Engineering Podcast

MARCH 10, 2024

However data engineers are challenged by both technical complexity and organizational complexity, with heterogeneous technologies to adopt, multiple data disciplines converging, legacy systems to support, and costs to manage. Links Project Nessie Article: What is Nessie, Catalog Versioning and Git-for-Data?

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. Workflow Definitions Below you can see a typical file structure of a sample workflow package written in SparkSQL. ??? backfill.sch.yaml ??? daily.sch.yaml ???

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Data Engineering Podcast

FEBRUARY 26, 2023

What are the technical systems that you are relying on to power the different data domains? What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Machine Learning Metadata Data Integration 147

The Pragmatic Engineer

SEPTEMBER 26, 2023

CDE vendors: a definite trend With more than 20 players in this space, let’s begin with a timeline of when the products launched. It’s also possible to posit that the “spark” which ignited this confluence of factors was the Covid-19 pandemic of 2020-21, and the accompanying rise of remote work.

Cloud 258

Cloud Software Engineer Software Engineering Cloud Computing 258

Christophe Blefari

JANUARY 20, 2024

Every company out there has his own definition for the data engineer role. What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Reddit r/dataengineering wiki a place where some data eng definitions are written. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

AUGUST 16, 2021

Users shouldn’t have to build their own central monitoring system. Before you can create any NiFi deployments with CDF-PC, you have to import your existing NiFi flow definitions into the Catalog. If you’re using the Apache NiFi Registry you can also export flow definitions from there that follow the same format. and later).

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Tweag

NOVEMBER 1, 2023

The last important remaining piece to explore is the merge system. In this post, we’ll see how to use the Nickel merge system to write reusable configuration modules, and why the merge approach seems to be more adapted for modular configurations than plain functions, despite Nickel being a functional language.

Metadata 113

Metadata Programming Programming Language Coding 113

Netflix Tech

MAY 19, 2023

By Chris Wolfe , Joey Schorr , and Victor Roldán Betancort Introduction The authorization team at Netflix recently sponsored work to add Attribute Based Access Control (ABAC) support to AuthZed’s open source Google Zanzibar inspired authorization system, SpiceDB. This would be a significant departure from its existing policy based system.

PostgreSQL 102

PostgreSQL Systems Accessible Accessibility 102

Knowledge Hut

MAY 3, 2024

Following is the authentic one-liner definition. Apache Spark is a fast and general-purpose, cluster computing system. One would find multiple definitions when you search the term Apache Spark. One would find the keywords ‘Fast’ and/or ‘In-memory’ in all the definitions. It was open-sourced in 2010 under a BSD license.

Hadoop 98

Hadoop Scala Healthcare Big Data 98

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the overall lifecycle of a feature, from definition to deployment and maintenance?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Max Kanat-Alexander and Grant Jenks Today we are open-sourcing the LinkedIn Developer Productivity & Happiness Framework (DPH Framework) - a collection of documents that describe the systems, processes, metrics, and feedback systems we use to understand our developers and their needs internally at LinkedIn.

Data Schemas 121

Data Schemas Software Engineer Software Engineering Data Pipeline 121

Engineering at Meta

OCTOBER 17, 2023

different data types stored in 21 different data systems. The third post will discuss SCARF’s orchestration for safely identifying and deleting unused data types across various data systems. The key implementation detail is the encoding of business logic that allows products and apps to function in Meta’s systems.

Coding 122

Coding Engineering Portfolio Data Schemas 122

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. We’ve used other metadata as well: doc is for documentation, while default indicates that the following definition is a default value. Venkatachalam, A. Chander, Z.

MySQL 134

MySQL Metadata Data Validation Coding 134

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. What is your working definition of "data governance" and how does that influence your product focus and priorities?

Data Governance 130

Data Governance Government Cloud Building 130

Christophe Blefari

MAY 18, 2023

Organisations are systems with inputs and outputs and a formula. Scaling Uber metrics systems (w/ Pinot) — uMetric migration from ES to Pinot. uMetric manages definition, discovery, computation, verification and serving. The analytics goal is to model correctly business.

Data 130

Data BI Consulting Data Science 130

The Pragmatic Engineer

APRIL 13, 2023

” My take is that in the way Covid-19 was an unforeseen ‘black swan’ event, so was the boom in tech and in VC-funding in 2021, which was definitely impacted by the pandemic, thanks to businesses and consumers shifting to digital, as a result of the lockdowns making in-person activities difficult and non-practical.

Finance 193

Finance Media Software Engineer Software Engineering 193

The Pragmatic Engineer

NOVEMBER 9, 2023

She asked the Director of Engineering if he could help take a known set of FAFSA application data and use it to artificially augment a much larger set of anonymous data tht her systems had collected over time. The engineering director’s next step?

Software Engineer 227

Software Engineer Software Engineering Engineering Coding 227

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

Cloudera

MAY 5, 2021

According to wikipedia , Gang Scheduling refers to a scheduling algorithm for parallel systems that schedules related threads or processes to run simultaneously on different processors. It mainly has 4 phases: Prepare : in this phase, YuniKorn proactively creates a certain number of placeholder pods based on the taskGroups definition.

Metadata 136

Metadata Algorithm Big Data Machine Learning 136

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In this blog post well talk about why a system like Glean is important, explain the rationale for Gleans design, and run through some of the ways were using Glean to supercharge our developer tooling at Meta.

Coding 55

Coding Programming Language SQL Programming 55