Tweag

NOVEMBER 1, 2023

If you control the source code, you can change the value provided to the configuration-as-a-function at the call-site. It also happens that one doesn’t control the call-site or the function’s code, because it’s in a library. In JSON, the best you can do is to copy and paste, praying for the different copies to not drift away with time.

Metadata 114

Metadata Programming Programming Language Coding 114

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

Monte Carlo

JULY 11, 2023

Data observability solutions also take testing a step further by providing root cause analysis tools that help teams proactively identify the root cause of data inaccuracies, as they relate to code, your data infrastructure, and even the data itself.

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40

Monte Carlo

AUGUST 30, 2023

More specifically, teams use dbt to write transformations as SQL queries, version control the code, and deploy transformations incrementally. If a test fails, i.e., the `dbt test` command exits with a non-zero code, you want your pipeline to halt and notify the team. With dbt tests, no news is good news. Curious to learn more?

SQL 52

SQL Datasets Database High Quality Data 52

Data Engineering Podcast

NOVEMBER 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

François Nguyen

JANUARY 18, 2021

This article was published in October 2020 with this title : “Data Governance as Code” The idea behind that is you should “actively promotes the safe use of data with automation that improves governance while freeing data analysts and scientists from manual tasks” The article is illustrated with many examples.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Monte Carlo

JULY 10, 2023

That’s where the definition of data completeness comes in. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. For example, address verification tools can supply missing ZIP codes when a customer neglects to provide theirs when filling out a form. The same is true with data.

Data Collection 40

Data Collection Data Governance Government Data 40

Data Engineering Podcast

JULY 24, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the definitions shifted over the past few decades? Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. It is easy to update the client code for web applications, but for the mobile native apps shipped to the app store, it is impossible to update the client code. We will discuss this in detail in the next section.

Metadata 82

Metadata Coding Banking Designing 82

François Nguyen

MARCH 22, 2021

This is really for us the definition of a self serve platform. ” Code : all the code necessary to build a data product (data pipelines, API, policies). Data As Code is a very strong choice : we do not want any UI because it is an heritage of the ETL period. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

SEPTEMBER 11, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What are the different places in a data system that schema definitions need to be established?

Systems 100

Systems Metadata Building MongoDB 100

Netflix Tech

JULY 22, 2024

By Jun He , Natallia Dzenisenka , Praneeth Yenugutala , Yingyi Zhang , and Anjali Norwood TL;DR We are thrilled to announce that the Maestro source code is now open to the public! A workflow definition is defined in a JSON format. Maestro supports dynamic parameters with code injection, which is super useful and powerful.

Machine Learning 102

Machine Learning Coding Data Pipeline Java 102

Data Engineering Podcast

MAY 22, 2022

In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful. What are the patterns that you and the community have established to encourage discovery and reuse of granular task definitions?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly. Snowflake Dynamic Tables.

Metadata 52

Metadata Data Lake Data Engineering Data Engineer 52

The Pragmatic Engineer

AUGUST 3, 2023

These links were especially helpful for: Promotions and marketing campaigns QR codes Content sharing links that “just work” Converting desktop users to mobile ones The shutdown Dynamic links powered Firebase Invites: an app invite service where users could send app invite links to their friends, to drive installation of the app.

Metadata 160

Metadata Engineering Building Technology 160

ThoughtSpot

JUNE 28, 2023

Applying analytics-as-code principles, we initially developed a scriptable approach with TML to automate the lifecycle management of analytic content across environments. Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses.

Metadata 97

Metadata Accessible Accessibility Coding 97

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Just like at first everyone had to code in a language, then everyone had to know how to incorporate packages from those languages — now we’re moving into, ‘ How do you incorporate AI that will write the code for you?’”

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

LinkedIn Engineering

APRIL 25, 2023

resource definitions, their entity schemas and their relationship definitions as input and generates a federated GraphQL type system. These data fetchers are written in a way to use the metadata available on our type system to self-configure and wire appropriately during the service startup.

Architecture 109

Architecture Metadata Java Transportation 109

Pinterest Engineering

OCTOBER 2, 2024

Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc., and also providing runtime metadata about the platform Security and Auditability , which ensures GDPR compliance, advanced levels of auditability, security, AAA, etc.

Data Management 56

Data Management Management Metadata SQL 56

Christophe Blefari

JUNE 16, 2023

Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI. The past often changes, whether because of code or data. This is metrics drift.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. How does it work?

BI 100

BI Data Warehouse Metadata SQL 100

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. Lack of unified metadata makes it difficult to optimize pipelines, avoid redundant processing, and quantify data engineering costs.

Project 52

Project Metadata Data Pipeline Data Engineering 52

ProjectPro

FEBRUARY 8, 2023

When Glue receives a trigger, it collects the data, transforms it using code that Glue generates automatically, and then loads it into Amazon S3 or Amazon Redshift. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. You can produce code, discover the data schema, and modify it. Why Use AWS Glue?

AWS 98

AWS Scala Metadata Data Lake 98

DoorDash Engineering

DECEMBER 5, 2023

Key terminology Here are definitions and links to further documentation about the tools we used. Terraform : Infrastructure-as-code (IaC) platform. Exposes a unique hardware definition language (HDL) for configuring compute resources. All Topics, even those without any messages, include some metadata that is backed up to disk.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

NOVEMBER 13, 2020

Metadata Caching. This is used to provide very low latency access to table metadata and file locations in order to avoid making expensive remote RPCs to services like the Hive Metastore (HMS) or the HDFS Name Node, which can be busy with JVM garbage collection or handling requests for other high latency batch workloads.

Metadata 142

Metadata Coding SQL Database 142

Towards Data Science

MAY 7, 2024

Now you have to decide what sorts of things to test in order to figure out definitively if the tool is the right commitment for you and the team. This will help create an overall picture of logging, metadata, latency, CDC, and all the other things that come with a pipeline. Gather the team together and determine what these things are.

ETL Tools 79

ETL Tools Metadata Data Integration Data Science 79

Data Engineering Podcast

MAY 18, 2021

Your host is Tobias Macey and today I’m interviewing Julien Le Dem about Open Lineage, a new standard for structuring metadata to enable interoperability across the ecosystem of data management tools. What is the current state of the ecosystem for generating and sharing metadata between systems?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Netflix Tech

JANUARY 10, 2024

Since then, the video pipeline has undergone substantial improvements and broad expansions: Starting with Standard Dynamic Range (SDR) at Standard-Definitions , we expanded the encoding pipeline to 4K and High Dynamic Range (HDR) which enabled support for our premium offering. This testing stage took about two weeks.

Process 93

Process Pipeline-centric Media Metadata 93

Engineering at Meta

OCTOBER 31, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. This is the third and final post in our series on Meta’s Systematic Code and Asset Removal Framework (SCARF). SCARF coordinates several kinds of tasks for each data system: metadata collection (e.g.,

Data 95

Data Metadata Coding Systems 95

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. With Alteryx , you can create workflows without needing to code by using the provided automation building blocks. Atlan Atlan data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 94

Metadata Python AWS Government 94

DataKitchen

FEBRUARY 2, 2023

As an analogy, the DevOps space has seen consolidation in code storage, CI/CD, team workflow, value stream management, testing, and other tools into one platform. It’s code -> governance, not governance->code. The code, configuration, and metadata about your data are the Intellectual Property of your data teams.

Metadata 52

Metadata BI Government Data Science 52

Monte Carlo

FEBRUARY 9, 2023

architecture (with some minor deviations) to achieve their data integration objectives around scalability and use of metadata. “A The other advantage is because we follow a standard design, we are able to generate a lot of our code using code templates and metadata. This layer has minimal transformation rules.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Monte Carlo

AUGUST 13, 2024

The Gartner definition of data observability Critical features of data observability What data observability monitors Comparing data quality vs data observability Considering embedded vs. standalone data observability solutions When do you need data observability? Let’s get into it. Table of Contents What is A Gartner Market Guide?

Data 52

Data Data Warehouse Data Pipeline Data Architecture 52

Netflix Tech

NOVEMBER 13, 2023

The detailed temporal metadata SMAD provides about speech and music regions in a polyphonic audio mixture are a first step for structural audio segmentation, indexing and pre-processing audio for the following downstream tasks. The heavy usage of ambient sounds and sound effects blurs the boundaries between active music regions and non-music.

Datasets 92

Datasets Metadata Algorithm Architecture 92

Towards Data Science

APRIL 6, 2023

Unit tests, which assess small pieces of code for performance on discrete, deterministic tasks, get overwhelmed by the variety of acceptable shapes and values that real-world data can take. That’s possibly correct — but the size of the distributional shift should definitely lead to an anomaly alert! What does each column represent?

Unstructured Data 78

Unstructured Data Metadata Data Coding 78