Maxime Beauchemin

AUGUST 28, 2017

Change Management Given that useful datasets become widely used and derived in ways that results in large and complex directed acyclic graphs (DAGs) of dependencies, altering logic or source data tends to break and/or invalidate downstream constructs. Upstream changes will inevitably break and invalidate downstream entities in intricate ways.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Databand.ai

SEPTEMBER 12, 2022

How to analyze dataset performance and schema changes in Databand Eric Jones 2022-09-12 13:06:42 “Why did my dataset schema change?” Databand helps fix this problem by capturing the metadata from your datasets and then alerting you when dataset operations change unexpectedly. Yeah, we hear this question a lot too.

Datasets 52

Datasets Metadata Data Engineering Data Engineer 52

Cloudera

SEPTEMBER 29, 2020

In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to Microsoft HDInsight (also powered by Apache Hive-LLAP) on Azure using the TPC-DS 2.9 A TPC-DS 10TB dataset was generated in ACID ORC format and stored on the ADLS Gen 2 cloud storage. benchmark.

Data Warehouse 121

Data Warehouse Cloud Storage Metadata Cloud 121

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Apache Iceberg is a high-performance open table format for petabyte-scale analytic datasets. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata.

Metadata 92

Metadata Data Warehouse BI AWS 92

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 119

Data Pipeline Building Manufacturing Data Warehouse 119

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 111

Data Warehouse Metadata Datasets Data 111

Netflix Tech

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. Configurability : TimeSeries offers a range of tunable options for each dataset, providing the flexibility needed to accommodate a wide array of use cases.

Bytes 95

Bytes Datasets Metadata Data 95

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 93

Datasets Metadata Algorithm Architecture 93

Netflix Tech

SEPTEMBER 18, 2024

message Item ( Bytes key, Bytes value, Metadata metadata, Integer chunk ) Database Agnostic Abstraction The KV abstraction is designed to hide the implementation details of the underlying database, offering a consistent interface to application developers regardless of the optimal storage system for that use case. . "persistence_configuration":[

Bytes 104

Bytes Metadata Database Data 104

Cloudera

AUGUST 10, 2021

For example, writing a Spark dataset to Ozone or launching a DDL query in Hive that points to a location in Ozone. I’ve chosen those names because I’ll be using an easy method for generating and writing TPC-DS datasets, along with creating their corresponding Hive tables. Create a dataset from the customer table. With CDP 7.1.4

Hadoop 106

Hadoop Kafka Datasets Government 106

Cloudera

SEPTEMBER 9, 2022

Iceberg is a next-generation, cloud-native table format designed to be open and scalable to petabyte datasets. With innovations like hidden partitioning and metadata stored at the file level, Iceberg makes querying on very large data sets faster, while also making changes to data easier and safer.

Architecture 103

Architecture Metadata Machine Learning Unstructured Data 103

Christophe Blefari

JULY 13, 2024

Fast News ⚡️ End-to-end data lineage in AWS — AWS announced DataZone to bring lineage to your data assets , from the picture it can mixes datasets (?), It provides abstractions and tools for the translation of lakehouse table format metadata. I'm not sure I'm happy to see this on Atlassian blog.

Kafka 130

Kafka AWS Data Database 130

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. Dataset execution is pipelined so that multiple execution stages can run in parallel.

Datasets 66

Datasets Software Engineer Software Engineering Metadata 66

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. In this blog post, we will highlight the work done recently to improve the performance of Ozone Manager to scale to exabytes of data. The hardware specifications are included at the end of this blog.

Management 71

Management Metadata Datasets Architecture 71

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 97

Management Kafka Metadata Media 97

Cloudera

MARCH 23, 2022

Therefore, alleviating the need to use different connectors, exotic and poorly maintained APIs, and other use-case specific workarounds to work with your datasets. . Iceberg is designed to be open and engine agnostic allowing datasets to be shared. 3: Open Performance.

Metadata 92

Metadata Data Architecture BI Machine Learning 92

ProjectPro

FEBRUARY 8, 2023

Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. For analyzing huge datasets, they want to employ familiar Python primitive types. billion by 2026?

AWS 98

AWS Scala Metadata Data Lake 98

Netflix Tech

MARCH 25, 2019

In the rest of this blog, we will a) touch on the complexity of Netflix cloud landscape, b) discuss lineage design goals, ingestion architecture and the corresponding data model, c) share the challenges we faced and the learnings we picked up along the way, and d) close it out with “what’s next” on this journey. push or pull.

Building 90

Building Metadata Transportation Data Ingestion 90

Data Engineering Podcast

AUGUST 5, 2019

Once a dataset has been located, how does Amundsen simplify the process of accessing that data for analysis or further processing? Links Amundsen Data Council Presentation Strata Presentation Blog Post Lyft Airflow Podcast.__init__ Links Amundsen Data Council Presentation Strata Presentation Blog Post Lyft Airflow Podcast.__init__

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 87

Metadata AWS Data Lake Datasets 87

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. These additional inputs are referred to as metadata filtering. Why use Confluent Cloud for Apache Flink and Rockset for RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Weekly

NOVEMBER 24, 2024

The blog highlights how moving from 6-character base-64 to 20-digit base-2 file distribution brings more distribution in S3 and reduces request failures. The blog is a good summary of how to use Snowflake QUERY_TAG to measure and monitor query performance. The blog post made me curious to understand DataFusion's internals.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Cloudera

DECEMBER 3, 2024

REST Catalog Value Proposition It provides open, metastore-agnostic APIs for Iceberg metadata operations, dramatically simplifying the Iceberg client and metastore/engine integration. It provides real time metadata access by directly integrating with the Iceberg-compatible metastore. You will see the 2 carrier records in the table.

Metadata 85

Metadata SQL Database Data Warehouse 85

ProjectPro

JANUARY 15, 2021

And honestly, there are a lot of real-world machine learning datasets around you that you can opt to start practicing your fundamental data science and machine learning skills, even without having to complete a comprehensive data science or machine learning course. Table of Contents What is a dataset in machine learning?

Machine Learning 40

Machine Learning Datasets Retail Banking 40

Cloudera

APRIL 3, 2023

In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services. Instead, Iceberg is intended for managing large, infrequently changing datasets.

Data Warehouse 83

Data Warehouse Java Metadata Data 83

Netflix Tech

NOVEMBER 20, 2023

The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset. In this blog post, we talk about the landscape and the challenges in workflows at Netflix. There are three common issues that the dataset owners usually face.

Process 87

Process Data Pipeline Datasets SQL 87

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. Streaming compute however, empowers more complex window queries on semantic correctness.

Big Data 86

Big Data Metadata Data Warehouse Data 86

Netflix Tech

MAY 5, 2021

Fig 1: Globally distributed production with artists working on different assets from different parts of the world There are workflows in which these artists may want to view a subset of these assets from this large dataset, for example, pertaining to a specific project. The major pieces, as shown in Fig. A sample manifest file.

Metadata 72

Metadata Bytes Media Cloud Storage 72

Airbnb Tech

NOVEMBER 28, 2023

In this blog post, we share our innovative approach to scoring data quality, Airbnb’s Data Quality Score (“DQ Score”). Luckily, we had two existing tools that would make this much easier: Dataportal (Airbnb’s data catalog and exploration UI), and the Unified Metadata Service (UMS).

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Databand.ai

AUGUST 30, 2023

Validity: Adherence to predefined formats, rules, or standards for each attribute within a dataset. Uniqueness: Ensuring that no duplicate records exist within a dataset. Integrity: Maintaining referential relationships between datasets without any broken links.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Cloudera

SEPTEMBER 24, 2020

However, there is a secret I am keeping to the end of the blog, which makes the decision even easier for the user: so easy in fact, you do not even have to decide yourself. Well, generally speaking, Impala works best when you are interacting with a data mart, which is typically a large dataset with a schema that is limited in scope.

Data Warehouse 72

Data Warehouse SQL Engineering Metadata 72

Netflix Tech

APRIL 26, 2022

An example of video data is video metadata, like the length of a video. These facts are managed and made available by services like viewing history or video metadata services outside of Axion. Now we asynchronously collect all the facts and metadata into a protobuf, compress it, and send it to the keystone messaging service.

Metadata 78

Metadata Datasets Machine Learning Designing 78

LinkedIn Engineering

MARCH 23, 2023

In this blog post, we will share our progress, challenges, and lessons learned from implementing Apache Beam. For batch processing, we use Spark to solve sophisticated batch scenarios and process petabytes of data with our industry-leading external shuffling service and schema metadata store.

Process 97

Process Lambda Architecture Kafka Datasets 97

Cloudera

JUNE 25, 2021

This blog post outlines detailed step by step instructions to perform Hive Replication from an on-prem CDH cluster to a CDP Public Cloud Data Lake. The Sentry service serves authorization metadata from the database backed storage; it does not handle actual privilege validation. This blog post is not a substitute for that.

Cloud 73

Cloud Data Lake Cloud Storage Metadata 73

phData: Data Engineering

DECEMBER 16, 2021

In blog 2 , we’ll explore: Who are the users of a data catalog? In blog 2 , we’ll explore: Who are the users of a data catalog? In blog 3 , we’ll look into: What data catalog options are available? A data catalog is a tool to store and access metadata about your data. What is Metadata?

Metadata 52

Metadata Datasets Data Accessibility 52

phData: Data Engineering

DECEMBER 16, 2021

If you missed our first blog on what a data catalog is , be sure to check it out! In this blog, we’ll explore what the ideal team to support your data catalog looks like. The Business Steward will be most familiar with the business drivers mentioned in the first blog in this series. How will Human-Driven Metadata be Captured?

Metadata 52

Metadata Datasets Data Education 52