Cloudera

MARCH 4, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 52

Metadata Data Governance Government Management 52

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. This is what managing data without metadata feels like. This is what managing data without metadata feels like. Effective metadata management is no longer a luxury—it’s a necessity.

Metadata 52

Metadata IT Government High Quality Data 52

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Metadata Caching. As Impala’s adoption grew the catalog service started to experience these growing pains, therefore recently we introduced two new features to alleviate the stress, On-demand Metadata and Zero Touch Metadata. More on this below.

Metadata 144

Metadata Coding SQL Database 144

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 134

MySQL Metadata Data Validation Coding 134

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

DECEMBER 3, 2022

All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command. This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Cloudera

FEBRUARY 22, 2022

Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . In CDP we enable Iceberg tables side-by-side with the Hive table types, both of which are part of our SDX metadata and security framework.

Metadata 111

Metadata Datasets BI SQL 111

Cloudera

AUGUST 10, 2021

We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 8: Data lineage based on Kafka Atlas Hook metadata. x, and 6.3.x,

Cloud 97

Cloud Kafka Metadata SQL 97

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 140

Algorithm Kafka Metadata Systems 140

Data Engineering Weekly

JUNE 16, 2024

The blog explains KIP-932 and its potential benefits. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Data Engineering Podcast

FEBRUARY 26, 2023

Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? TimeXtender Logo]([link] TimeXtender is a holistic, metadata-driven solution for data integration, optimized for agility. When is a data mesh the wrong choice?

Building 147

Building Machine Learning Metadata Data Integration 147

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Metadata 112

Metadata Government Data Governance Data 112

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 102

Building Metadata Cloud Storage AWS 102

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Given our role on this critical path, accuracy is paramount. Psyberg: The Game Changer!

Data Engineering 108

Data Engineering Data Engineer Engineering Metadata 108

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Metadata 93

Metadata Data Warehouse BI AWS 93

Cloudera

JULY 10, 2024

Observability for your most secure data For your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment. The post Introducing Cloudera Observability Premium appeared first on Cloudera Blog.

Metadata 101

Metadata Cloud Management IT 101

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. A comprehensive list of benefits offered by Cosmos can be found in the linked blog.

Process 95

Process Pipeline-centric Media Metadata 95

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 94

Data Process Process Metadata Finance 94

Data Engineering Weekly

JUNE 24, 2024

The blog contains a summary of each talk and a link to the YouTube channel with all the talks. The blog details the classification model, training approach and historical data analysis. Influx data conclude that while technical concerns about Parquet metadata are valid, the actual overhead is smaller than generally recognized.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Netflix Tech

MARCH 10, 2023

We built an asset management platform (AMP), codenamed Amsterdam , in order to easily organize and manage the metadata, schema, relations and permissions of these assets. In this blog, we will be focusing on how we utilize Elasticsearch for indexing and search the assets. This is the layer we’d like to focus on in this blog.

Management 106

Management Metadata Digital Media Kafka 106

Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. Parquet also stores type metadata which makes reading back and processing the files later slightly easier. Introduction. The raw data is in a series of CSV files. Register Now. .

Machine Learning 138

Machine Learning Datasets Data Science Raw Data 138

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 101

Metadata Data Pipeline Scala Data Workflow 101

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. The example 1_typedef-server.json describes the server typedef used in this blog. .

Data Governance 98

Data Governance Government Metadata Datasets 98

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. More details on the research behind the Cortex Search retrieval stack will be shared on our Snowflake Engineering Blog.

Unstructured Data 114

Unstructured Data Metadata Government SQL 114

Cloudera

SEPTEMBER 29, 2020

In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to Microsoft HDInsight (also powered by Apache Hive-LLAP) on Azure using the TPC-DS 2.9 The post Cloudera Data Warehouse outperforms Azure HDInsight in TPC-DS benchmark appeared first on Cloudera Blog.

Data Warehouse 122

Data Warehouse Cloud Storage Metadata Cloud 122

DoorDash Engineering

MARCH 14, 2023

At DoorDash, we view each failure as a learning opportunity and sometimes share our insights and lessons learned in public blog posts to show our commitment to reliability and knowledge sharing. Each section is accompanied by real outages drawn from our past blog posts that can be explored in greater detail.

Java 135

Java Metadata Database Systems 135

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 111

Data Warehouse Metadata Datasets Data 111

Snowflake

AUGUST 7, 2024

As we describe in this blog post , the top-k feature uses runtime information — namely, the current contents of the top-k elements — to skip micro-partitions where we can guarantee that they won’t contribute to the overall result. on average, with some queries also reaching up to 99.8% improvement. How does it work?

Metadata 81

Metadata Algorithm Process Utilities 81

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

SEPTEMBER 2, 2020

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. In an earlier blog post, we discussed Telltale , our health monitoring system. by Elizabeth Carretto Everyone loves Unsolved Mysteries. What is Edgar?

Metadata 119

Metadata Transportation Engineering Systems 119

Cloudera

APRIL 2, 2024

Our use of metadata also ensures persistent context across all analytics and cloud environments, making it easier to maintain consistent security policies and access controls. The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

Designing 103

Designing Process Government Metadata 103

Netflix Tech

JANUARY 25, 2023

For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. In Marken, an annotation is a piece of metadata which can be attached to an object from any domain. Actual data is stored in the metadata section of json. In this case it is BOUNDING_BOX.

Algorithm 116

Algorithm Media Metadata Data Ingestion 116

Cloudera

SEPTEMBER 9, 2022

With innovations like hidden partitioning and metadata stored at the file level, Iceberg makes querying on very large data sets faster, while also making changes to data easier and safer. So your data never moves, by just changing your metadata you can start benefiting from the Iceberg table format immediately. .

Architecture 104

Architecture Metadata Machine Learning Unstructured Data 104

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 120

Data Pipeline Building Manufacturing Data Warehouse 120

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications.

Architecture 109

Architecture Metadata Java Transportation 109