Machine Learning Metadata Store
KDnuggets
AUGUST 31, 2022
In this article, we will learn about metadata stores, the need for them, their components, and metadata store management.
Metadata Related Topics 
How Metadata Improves Security, Quality, and Transparency
KDnuggets
APRIL 25, 2022
Metadata is the data providing context about the data, more than what you see in the rows and columns. By managing your metadata, you're effectively creating an encyclopedia of your data assets.
Metadata 
156
156 
Data Engineering Best Practices - #2. Metadata & Logging
Start Data Engineering
FEBRUARY 22, 2024
Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Data Pipeline Logging Best Practices 3.1. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.
Metadata 130
130 
Level Up Your Data Platform With Active Metadata
Data Engineering Podcast
JUNE 19, 2022
Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.
Metadata 130
130 
Metadata – Data Interoperability’s Hidden Talent (Part Two)
ArcGIS
SEPTEMBER 23, 2024
Metadata, the data about your data, is incredibly important, and Data Interoperability can help you create, manage, and maintain that data.
Metadata 110
110 
Metadata – Data Interoperability’s Hidden Talent (Part One)
ArcGIS
SEPTEMBER 23, 2024
Metadata, the data about your data, is incredibly important, and Data Interoperability can help you create, manage, and maintain that data.
Metadata 98
98 
Octopai Acquisition Enhances Metadata Management to Trust Data Across Entire Data Estate
Cloudera
NOVEMBER 13, 2024
It leverages knowledge graphs to keep track of all the data sources and data flows, using AI to fill the gaps so you have the most comprehensive metadata management solution. Together, Cloudera and Octopai will help reinvent how customers manage their metadata and track lineage across all their data sources.
Metadata 96
96 
Interesting startup idea: benchmarking cloud platform pricing
The Pragmatic Engineer
OCTOBER 17, 2024
Results are stored in git and their database, together with benchmarking metadata. Benchmarking results for each instance type are stored in sc-inspector-data repo, together with the benchmarking task hash and other metadata. There Then we wait for the actual data and/or final metadata (e.g.
Cloud 278
278 
Data News — Week 24.11
Christophe Blefari
MARCH 15, 2024
Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.
Metadata 272
272 
Using Images and Metadata for Product Fuzzy Matching with Zingg
databricks
SEPTEMBER 24, 2023
Product matching is an essential function in many retail and consumer goods organizations. Incoming products are compared to items in the existing product.
Metadata 110
110 
How to get started with dbt
Christophe Blefari
MARCH 1, 2023
You can also add metadata on models (in YAML). docs — in dbt you can add metadata on everything, some of the metadata is already expected by the framework and thank to it you can generate a small web page with your light catalog inside: you only need to do dbt docs generate and dbt docs serve.
Data Warehouse 278
278 
Tracking Schema Changes in Iceberg Tables Using Metadata Files
Cloudyard
OCTOBER 15, 2024
When using Iceberg tables, every Data Definition Language ( DDL ) operation triggers the generation of a new metadata JSON file that captures the updated structure. This article outlines a process for efficiently tracking schema changes in Iceberg tables by leveraging Snowflake’s powerful metadata storage capabilities.
Metadata 52
52 
Databricks, Snowflake and the future
Christophe Blefari
JUNE 21, 2024
Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. That's why you need a catalog.
Metadata 147
147 
Being Data Driven At Stripe With Trino And Iceberg
Data Engineering Podcast
JUNE 16, 2024
what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? What were the requirements and selection criteria that led to the selection of that combination of technologies?
Data Lake 147
147 
Keeping Small Queries Fast – Short query optimizations in Apache Impala
Cloudera
NOVEMBER 13, 2020
Metadata Caching. This is used to provide very low latency access to table metadata and file locations in order to avoid making expensive remote RPCs to services like the Hive Metastore (HMS) or the HDFS Name Node, which can be busy with JVM garbage collection or handling requests for other high latency batch workloads.
Metadata 144
144 
Reflecting On The Past 6 Years Of Data Engineering
Data Engineering Podcast
FEBRUARY 5, 2023
Orchestration is now a part of most vertical tools Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything Data catalog -> data discovery -> active metadata Business intelligence Read only reports to metric/semantic layers Embedded analytics and data APIs Rise of ELT dbt Corresponding introduction (..)
Data Engineering 147
147 
Ready-to-go sample data pipelines with Dataflow
Netflix Tech
DECEMBER 3, 2022
This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps. DDL Often, the first step in a data pipeline is to define the target table structure and column metadata via a DDL statement. For the workflow orchestration we use Netflix homegrown Maestro scheduler.
Data Pipeline 144
144 
Build Data Products Without A Data Team Using AgileData
Data Engineering Podcast
NOVEMBER 13, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Building 130
130 
The Data Discovery Team
Jesse Anderson
NOVEMBER 14, 2023
That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.
Metadata 147
147 
Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas
Data Engineering Podcast
JUNE 19, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Metadata 130
130 
What do Snowflake, Databricks, Redshift, BigQuery actually do?
Start Data Engineering
NOVEMBER 21, 2024
Metadata catalog stores information about datasets 3.1.3. Most platforms enable you to do the same thing but have different strengths 3.1. Understand how the platforms process data 3.1.1. A compute engine is a system that transforms data 3.1.2. Data platform support for SQL, Dataframe, and Dataset APIs 3.1.4.
Metadata 130
130 
A Look At The Data Systems Behind The Gameplay For League Of Legends
Data Engineering Podcast
NOVEMBER 20, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. From analyzing your metadata, query logs, and dashboard activities, Select Star will automatically document your datasets.
Systems 130
130 
Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform
Data Engineering Podcast
JUNE 26, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Datasets 130
130 
Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense
Data Engineering Podcast
OCTOBER 30, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Build Better Data Products By Creating Data, Not Consuming It
Data Engineering Podcast
NOVEMBER 6, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Building 130
130 
Making Sense Of The Technical And Organizational Considerations Of Data Contracts
Data Engineering Podcast
DECEMBER 18, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!
Metadata 130
130 
Announcing Nickel 1.0
Tweag
MAY 16, 2023
Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.
MySQL 134
134 
Making The Total Cost Of Ownership For External Data Manageable With Crux
Data Engineering Podcast
JULY 17, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Data Management 130
130 
Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI
Data Engineering Podcast
DECEMBER 29, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!
Management 130
130 
Accelerate Your Machine Learning Workflows in Snowflake with Snowpark ML
Snowflake
JANUARY 23, 2024
Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.
Machine Learning 123
123 
How To Bring Agile Practices To Your Data Projects
Data Engineering Podcast
OCTOBER 23, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Project 130
130 
The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse
Data Engineering Podcast
FEBRUARY 19, 2023
Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform. Acryl]([link] The modern data stack needs a reimagined metadata management platform.
IT 147
147 
Increase data literacy and trust with Alation data catalog integration
ThoughtSpot
OCTOBER 9, 2023
How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.
Metadata 105
105 
1. Streamlining Membership Data Engineering at Netflix with Psyberg
Netflix Tech
NOVEMBER 14, 2023
It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities. All Iceberg tables have associated metadata that provide insight into changes or updates within the data tables.
Data Engineering 109
109 
Logarithm: A logging engine for AI training workflows and services
Engineering at Meta
MARCH 18, 2024
Users can query using regular expressions on log lines, arbitrary metadata fields attached to logs, and across log files of hosts and services. Each log line can have zero or more metadata key-value pairs attached to it. The extracted key-value pairs are added to the log line’s metadata. in PyTorch). Multimodal data (e.g.,
Engineering 106
106 
Data Engineering Best Practices - #1. Data flow & Code
Start Data Engineering
JULY 20, 2023
Know the when, how, & what (aka metadata) of pipeline runs for easier debugging 3. Ensure data is valid before exposing it to its consumers (aka data quality checks) 3.3. Avoid data duplicates with idempotent pipelines 3.4. Write DRY code & keep I/O separate from data transformation 3.5.
Coding 130
130 
Why Spatial Data Governance is Critical to Your Business Strategy
Precisely
NOVEMBER 14, 2023
This journey must include a strong data governance framework to align people, processes, and technology, and enable them to understand and trust their data and metadata to achieve their business objectives. Does our organization’s data governance service include visibility and transparency of our spatial data and their metadata?
Data Governance 119
119 
Video Multiplexer Tips and Tricks
ArcGIS
APRIL 16, 2024
Tips to properly format your metadata for the video multiplexer tool so you can geoenable video data for the Full Motion Video player.
Metadata 111
111 
Snowflake Invests in Metaplane for Deep, End-to-End Observability in the Data Cloud
Snowflake
MAY 15, 2024
By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly. Metaplane ensures that every company can trust the data that powers their business.
Cloud 114
114 
Data Engineering Weekly #176
Data Engineering Weekly
JUNE 16, 2024
[link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.
Data Engineering 105
105 
Modern Data Architecture: Data Mesh and Data Fabric 101
Precisely
OCTOBER 31, 2024
While data products may have different definitions in different organizations, in general it is seen as data entity that contains data and metadata that has been curated for a specific business purpose. A data fabric weaves together different data management tools, metadata, and automation to create a seamless architecture.
Build an Open Data Lakehouse with Iceberg Tables, Now in Public Preview
Snowflake
DECEMBER 4, 2023
With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?
Building 103
103 
Introducing Cloudera Observability Premium
Cloudera
JULY 10, 2024
Observability for your most secure data For your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment.
Metadata 105
105 
Ensuring the Successful Launch of Ads on Netflix
Netflix Tech
JUNE 1, 2023
It also included metadata about ads, such as ad placement and impression-tracking events. A Kafka consumer retrieved the playback manifests with ad metadata and simulated a device playing the content and triggering the impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.
Algorithm 140
140 
Let's personalize your content