Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 120

Building Metadata Cloud Storage AWS 120

dbt Developer Hub

APRIL 20, 2025

These tools can be called by LLM systems to learn about your data and metadata. and receive accurate information based on your dbt project's documentation and structure. The dbt MCP server provides access to a set of tools that operate on top of your dbt project. Business users can ask questions like "What customer data do we have?"

Structured Data 98

Structured Data SQL BI Project 98

Netflix Tech

DECEMBER 3, 2022

For more information on this and other examples please visit the Dataflow documentation page." This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps. Attributes are set via Metacat , which is a Netflix internal metadata management platform.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Data Engineering Podcast

NOVEMBER 20, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Select Star is a data discovery platform that automatically analyzes & documents your data.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Jesse Anderson

NOVEMBER 14, 2023

That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.

Metadata 147

Metadata Data Science Big Data Data 147

Snowflake

JANUARY 23, 2024

Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.

Machine Learning 137

Machine Learning Metadata Python Telecommunication 137

Rockset

JANUARY 23, 2024

When building applications on change data capture (CDC) data using Elasticsearch, you’ll want to architect the system to handle frequent updates or modifications to the existing documents in an index. When a user searches for a show, ie “political thriller”, they are returned a set of relevant results based on keywords and other metadata.

Metadata 52

Metadata Coding Analytics Application Python 52

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 135

MySQL Metadata Coding Data Validation 135

Netflix Tech

MARCH 10, 2023

We built an asset management platform (AMP), codenamed Amsterdam , in order to easily organize and manage the metadata, schema, relations and permissions of these assets. It provides simple APIs for creating indices, indexing or searching documents, which makes it easy to integrate. are stored in secure storage layers.

Management 106

Management Metadata Digital Media Kafka 106

Snowflake

JUNE 11, 2024

The Snowflake Model Registry , in general availability, provides a centralized repository to manage all models and their related artifacts and metadata. Teams can visually interact with Feature Store objects and their metadata from a new UI (private preview) in Snowsight. Ask the Community!

Management 112

Management Government Metadata Python 112

Cloudera

SEPTEMBER 28, 2020

Review the Upgrade document topic for the supported upgrade paths. Document the number of dev/test/production clusters. Document the operating system versions, database versions, and JDK versions. Review the JDK versions and determine if a JDK version change is needed and if so follow the documentation here to upgrade.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Data Engineering Weekly

APRIL 6, 2025

One reason why all the engineering documentation fails and quickly becomes outdated is that it is always written from the author's perspective. Unlike coding, we never (or rarely) apply a code review process for documentation. [link] Grab: Facilitating Docs-as-Code implementation for users unfamiliar with Markdown.

Data Engineer 62

Data Engineer Data Engineering Engineering Kafka 62

François Nguyen

MARCH 22, 2021

and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…). The data domain Discovery portal with all the metadata on the data life cycle 4.Federated TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality.

Technology 130

Technology Architecture Google Cloud Metadata 130

Cloudera

NOVEMBER 13, 2020

Metadata Caching. This is used to provide very low latency access to table metadata and file locations in order to avoid making expensive remote RPCs to services like the Hive Metastore (HMS) or the HDFS Name Node, which can be busy with JVM garbage collection or handling requests for other high latency batch workloads. Next Steps.

Metadata 143

Metadata Coding SQL Database 143

Netflix Tech

JUNE 1, 2023

The replay traffic environment generated responses containing a standard playback manifest, a JSON document containing all the necessary information for a Netflix device to start playback. It also included metadata about ads, such as ad placement and impression-tracking events.

Algorithm 140

Algorithm Kafka Metadata Systems 140

Snowflake

APRIL 11, 2024

Not only do we have a unique vantage point into the challenges faced by data analysts, we also possess rich metadata that feeds into Snowflake’s dedicated text-to-SQL model that Copilot leverages in combination with Mistral’s technology. This vast amount of data fuels the development of Copilot, surpassing typical large language models.

SQL 140

SQL Data Analysis AWS High Quality Data 140

Data Engineering Weekly

JUNE 16, 2024

The official design document for liquid clustering is here. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Snowflake

NOVEMBER 2, 2023

For example, customers who need a centralized store of data in large volume and variety – including JSON, text files, documents, images, and video – have built their data lake with Snowflake. For public preview or generally available features, please read the release notes and documentation to learn more and get started.

Data Lake 114

Data Lake Data Warehouse Cloud Unstructured Data 114

Maxime Beauchemin

JANUARY 20, 2017

This opens new moves in the data modeler’s playbook, and can allow for fact tables to store multiple grains at once when needed dynamic schemas : since the advent of map reduce, with the growing popularity of document stores and with support for blobs in databases, it’s becoming easier to evolve database schemas without executing DML.

Data Engineer 130

Data Engineer Data Engineering Engineering ETL Tools 130

Rockset

APRIL 18, 2023

In the demo, you’ll see how Rockset delivers search results in 15 milliseconds over thousands of documents. Organizations have continued to accumulate large quantities of unstructured data, ranging from text documents to multimedia content to machine and sensor data. Why use vector search?

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Snowflake

JUNE 4, 2024

Generative AI presents enterprises with the opportunity to extract insights at scale from unstructured data sources, like documents, customer reviews and images. Cortex Search can scale to millions of documents with subsecond latency, using fully managed vector embedding and retrieval.

Data Security 119

Data Security Machine Learning Unstructured Data SQL 119

Engineering at Meta

FEBRUARY 6, 2024

In this way, DotSlash simplifies the work of cross-platform releases: In this example, the workflow DotSlash runs through when executing node looks like: See the How DotSlash Works documentation for details. See the Generating DotSlash Files at Meta documentation for details. Because of how #! node --version.

Metadata 127

Metadata Coding Building Project 127

Tweag

NOVEMBER 1, 2023

We can extract documentation from them, get completion in the LSP, use nickel query , and so on. Metadata can be attached to record fields, giving them more expressive power and the ability to better describe an interface for a partial configuration. You need to provide arguments first. But query doesn’t work on functions!

Metadata 117

Metadata Programming Programming Language Coding 117

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Snowflake

AUGUST 7, 2024

Before Snowflake starts executing the query, we look at the metadata of the partitions to determine whether the contents of a given partition are likely to end up in the final result. For a list of key performance improvements by year and month, visit Snowflake Documentation. Snowflake starts processing those partitions first.

Metadata 99

Metadata Algorithm Process Utilities 99

Lyft Engineering

JUNE 28, 2023

The interface was designed such that a minimal amount of metadata was needed to construct a pipeline object which performs a given capability. The steep learning curve for developers when using streaming which required detailed documentation. We called this the Open Beta phase of the project.

Machine Learning 122

Machine Learning Building Kafka Metadata 122

Monte Carlo

DECEMBER 12, 2024

It means defining that data by documenting relationships between creator and context (like customers and their orders), establishing clear business definitions (what exactly counts as an “active user”?), and maintaining metadata about data freshness, quality, and lineage (more on that in a moment).

Government 52

Government Data Cloud Datasets 52

Christophe Blefari

JUNE 16, 2023

Data Documentation 101: Why? — Marie wrote best practices for establishing complete and reliable data documentation. The first advice is about the documentation readers: data team, business users or other stakeholders. Data warehouses are mutable, this is one of the many root causes proposed by Lucas.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

ProjectPro

FEBRUARY 8, 2023

Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features. Why Use AWS Glue? being data exactly matches the classifier, and 0.0 doesn't match the classifier.

AWS 98

AWS Scala Metadata Data Lake 98

Cloudera

OCTOBER 7, 2021

Data and Metadata: Data inputs and data outputs produced based on the application logic. Also included, business and technical metadata, related to both data inputs / data outputs, that enable data discovery and achieving cross-organizational consensus on the definitions of data assets.

Architecture 124

Architecture Metadata Kafka Government 124

Cloudera

AUGUST 10, 2021

Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. x, as well as documented rollback procedures to help customers with their move to CDP PvC Base, as mentioned in the blog introduction. .

Cloud 97

Cloud Kafka Metadata SQL 97

Snowflake

AUGUST 22, 2023

Finally, not to be overlooked are the metadata and documentation required to ensure the product can easily be used. Metadata and data should be easy to find for both humans and computers. Machine-readable metadata is essential for automatic discovery of data sets and services.

Metadata 94

Metadata Data AWS Business Analyst 94

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets.

Data Governance 97

Data Governance Government Metadata Datasets 97

ThoughtSpot

JUNE 28, 2023

Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses. For more information, explore our developer documentation on the new version control REST APIs for Git. vcs/git/config/create REST API v2.0 We'd love to hear your feedback.

Metadata 97

Metadata Accessibility Accessible Coding 97

Data Engineering Podcast

NOVEMBER 27, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. images, documents, etc.) Atlan is the metadata hub for your data ecosystem. images, documents, etc.) Stale dashboards?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Picnic Engineering

FEBRUARY 6, 2025

Expanding this type-based schema with some additional metadata allowed us to autogenerate the UI for whatever configuration parameters a component needs. To do so, we generalized what we alreadyhad: The components that we built already had a schema defining what input they needed. To configure pages, we already had our own DSL.

Metadata 52

Metadata Architecture SQL Building 52

Cloudera

MARCH 31, 2021

Customers who have chosen Google Cloud as their cloud platform can now use CDP Public Cloud to create secure governed data lakes in their own cloud accounts and deliver security, compliance and metadata management across multiple compute clusters. Google Cloud Storage buckets – in the same subregion as your subnets .

Google Cloud 94

Google Cloud Cloud Amazon Web Services Cloud Storage 94