Data Engineering Podcast

AUGUST 24, 2020

The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub. What were you using at LinkedIn for metadata management prior to the introduction of DataHub?

Metadata 100

Metadata Management Kafka Data Engineering 100

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

dbt Developer Hub

SEPTEMBER 14, 2021

Give Jeremy a win and check out the blog he just posted on why this matters even more leading up to ?dbt Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! While we're throwing W's, don't forget to also register for his talk at Coalesce now !

Metadata 52

Metadata Data Pipeline Python Cloud 52

Jesse Anderson

NOVEMBER 14, 2023

A Guest Post by Ole Olesen-Bagneux In this blog post I would like to describe a new data team, that I call ‘the data discovery team’. And that is what the data discovery team that I propose in this blog post should work on: searching for data. In an enterprise data reality, searching for data is a bit of a hassle.

Metadata 147

Metadata Data Science Big Data Data 147

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. So which static site generator to choose?

Engineering 40

Engineering Bytes AWS Python 40

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table.

Metadata 147

Metadata Data Warehouse BI Scala 147

Snowflake

AUGUST 7, 2024

As we describe in this blog post , the top-k feature uses runtime information — namely, the current contents of the top-k elements — to skip micro-partitions where we can guarantee that they won’t contribute to the overall result. on average, with some queries also reaching up to 99.8% improvement. How does it work?

Metadata 61

Metadata Algorithm Process Utilities 61

Data Engineering Weekly

JUNE 16, 2024

The blog explains KIP-932 and its potential benefits. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Cloudera

JULY 10, 2024

Observability for your most secure data For your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment. The post Introducing Cloudera Observability Premium appeared first on Cloudera Blog.

Metadata 98

Metadata Cloud Management IT 98

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 99

Building Metadata Cloud Storage AWS 99

Cloudera

NOVEMBER 15, 2024

The metadata-driven approach ensures quick query planning so defenders don’t have to deal with slow processes when they need fast answers. Iceberg makes query planning more efficient by storing all of the table metadata–including partitioning and file locations–in a way that’s easy for query engines to consume.

Metadata 77

Metadata Unstructured Data Data Lake Government 77

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 92

Data Process Process Metadata Finance 92

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Metadata 90

Metadata Data Warehouse BI AWS 90

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Given our role on this critical path, accuracy is paramount. Psyberg: The Game Changer!

Data Engineering 107

Data Engineering Data Engineer Engineering Metadata 107

Snowflake

AUGUST 7, 2024

As we describe in this blog post , the top-k feature uses runtime information — namely, the current contents of the top-k elements — to skip micro-partitions where we can guarantee that they won’t contribute to the overall result. on average, with some queries also reaching up to 99.8% improvement. How does it work?

Metadata 52

Metadata Algorithm Utilities Architecture 52

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. Results are from vLLM team’s blog.

Datasets 63

Datasets Software Engineer Software Engineering Metadata 63

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Metadata 110

Metadata Government Data Governance Data 110

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. A comprehensive list of benefits offered by Cosmos can be found in the linked blog.

Process 93

Process Pipeline-centric Media Metadata 93

Pinterest Engineering

OCTOBER 2, 2024

Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc., and also providing runtime metadata about the platform Security and Auditability , which ensures GDPR compliance, advanced levels of auditability, security, AAA, etc.

Data Management 56

Data Management Management Metadata SQL 56

Data Engineering Weekly

JUNE 24, 2024

The blog contains a summary of each talk and a link to the YouTube channel with all the talks. The blog details the classification model, training approach and historical data analysis. Influx data conclude that while technical concerns about Parquet metadata are valid, the actual overhead is smaller than generally recognized.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. These additional inputs are referred to as metadata filtering.

Cloud 64

Cloud Building Metadata Kafka 64

Data Council

MARCH 25, 2021

How to Build a Community (Fishtown Analytics) Claire Carroll's first personal blog post on community-building is a must-read. Here's our March 2021 roundup of links from across the web that we selected for you: 1. As Fishtown Analytics' community manager for the last 2.5

Metadata 52

Metadata Management Building IT 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 99

Metadata Data Pipeline Scala Data Workflow 99

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 135

MySQL Metadata Data Validation Coding 135

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. More details on the research behind the Cortex Search retrieval stack will be shared on our Snowflake Engineering Blog.

Unstructured Data 111

Unstructured Data Metadata Government SQL 111

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. The data files and metadata files in Iceberg format are immutable.

Java 70

Java Metadata PostgreSQL Data Warehouse 70

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. In this blog post, we will highlight the work done recently to improve the performance of Ozone Manager to scale to exabytes of data. The hardware specifications are included at the end of this blog.

Management 70

Management Metadata Datasets Architecture 70

Databand.ai

AUGUST 30, 2023

Learn more in our detailed guide to data reliability testing Metadata Management Metadata management is the process of organizing, maintaining, and using metadata to improve the quality, consistency, and usability of your data.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Cloudera

OCTOBER 10, 2024

Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance. The post Cloudera Lakehouse Optimizer Makes it Easier Than Ever to Deliver High-Performance Iceberg Tables appeared first on Cloudera Blog.

IT 84

IT Data Lake Metadata Data Warehouse 84

Netflix Tech

SEPTEMBER 18, 2024

message Item ( Bytes key, Bytes value, Metadata metadata, Integer chunk ) Database Agnostic Abstraction The KV abstraction is designed to hide the implementation details of the underlying database, offering a consistent interface to application developers regardless of the optimal storage system for that use case. . "persistence_configuration":[

Bytes 99

Bytes Metadata Database Data 99

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. A proper survey of these approaches would fill many blog posts of this size. The same applies to the metadata of your system.

Metadata 52

Metadata Database Algorithm SQL 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

phData: Data Engineering

NOVEMBER 8, 2024

In this blog, we will discuss: What is the Open Table format (OTF)? Then, we add another column called HASHKEY , add more data, and locate the S3 file containing metadata for the iceberg table. In the screenshot below, we can see that the metadata file for the Iceberg table retains the snapshot history. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52