DataKitchen

FEBRUARY 20, 2025

Now With Actionable, Automatic, Data Quality Dashboards Imagine a tool that can point at any dataset, learn from your data, screen for typical data quality issues, and then automatically generate and perform powerful tests, analyzing and scoring your data to pinpoint issues before they snowball. Announcing DataOps Data Quality TestGen 3.0:

Datasets 73

Datasets Metadata Data Government 73

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. In this blog, we will discuss: What is the Open Table format (OTF)? These formats are transforming how organizations manage large datasets. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Netflix Tech

FEBRUARY 14, 2025

In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. Architecture Overview The first pivotal step in managing impressions begins with the creation of a Source-of-Truth (SOT) dataset.

Kafka 66

Kafka Datasets Metadata Utilities 66

Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. To try and predict this, an extensive dataset including anonymised details on the individual loanee and their historical credit history are included. Get the Dataset. Introduction.

Machine Learning 137

Machine Learning Data Science Datasets Raw Data 137

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Data Engineering Weekly

APRIL 6, 2025

The Grab blog delights me since I have tried to do this many times. A cross-encoder teacher model, fine-tuned on human-labeled data and enriched Pin metadata, was distilled into a lightweight student model using semi-supervised learning over billions of impressions. Kudos to the Grab team for building a docs-as-code system.

Data Engineer 63

Data Engineer Data Engineering Engineering Kafka 63

Cloudera

FEBRUARY 22, 2022

The data architecture layer is one such area where growing datasets have pushed the limits of scalability and performance. The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg.

Metadata 110

Metadata Datasets BI SQL 110

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Embrace Version Control for Data and Code: Just as software developers use version control for code, DataOps involves tracking versions of datasets and data transformation scripts.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? So clearly Impala is used extensively with datasets both small and large. Metadata Caching. More on this below.

Metadata 143

Metadata Coding SQL Database 143

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. Kafka: Kafka stores metadata about connectors in several internal topics that are not exposed to end users. What is Change Data Capture? or its affiliates.

Kafka 50

Kafka MySQL Database Software Engineering 50

Netflix Tech

DECEMBER 3, 2022

mock Generate or validate mock datasets. All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command. " ) COMMENT "Example dataset brought to you by Dataflow. -v, --verbose Enables verbose mode. version Show the version and exit.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. The example 1_typedef-server.json describes the server typedef used in this blog. .

Data Governance 97

Data Governance Government Metadata Datasets 97

Databand.ai

SEPTEMBER 12, 2022

How to analyze dataset performance and schema changes in Databand Eric Jones 2022-09-12 13:06:42 “Why did my dataset schema change?” Databand helps fix this problem by capturing the metadata from your datasets and then alerting you when dataset operations change unexpectedly. Yeah, we hear this question a lot too.

Datasets 52

Datasets Metadata Data Engineering Data Engineer 52

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Embrace Version Control for Data and Code: Just as software developers use version control for code, DataOps involves tracking versions of datasets and data transformation scripts.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Maxime Beauchemin

AUGUST 28, 2017

Change Management Given that useful datasets become widely used and derived in ways that results in large and complex directed acyclic graphs (DAGs) of dependencies, altering logic or source data tends to break and/or invalidate downstream constructs. Upstream changes will inevitably break and invalidate downstream entities in intricate ways.

Data Engineer 174

Data Engineer Data Engineering Engineering Software Engineering 174

Cloudera

SEPTEMBER 29, 2020

In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to Microsoft HDInsight (also powered by Apache Hive-LLAP) on Azure using the TPC-DS 2.9 A TPC-DS 10TB dataset was generated in ACID ORC format and stored on the ADLS Gen 2 cloud storage. benchmark.

Data Warehouse 120

Data Warehouse Cloud Storage Metadata Cloud 120

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 110

Data Warehouse Metadata Datasets Machine Learning 110

Netflix Tech

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. Configurability : TimeSeries offers a range of tunable options for each dataset, providing the flexibility needed to accommodate a wide array of use cases.

Bytes 95

Bytes Datasets Metadata Data 95

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 105

Metadata AWS Data Lake Datasets 105

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. Dataset execution is pipelined so that multiple execution stages can run in parallel.

Datasets 71

Datasets Software Engineering Software Engineer Metadata 71

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Apache Iceberg is a high-performance open table format for petabyte-scale analytic datasets. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata.

Metadata 91

Metadata Data Warehouse BI AWS 91

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 93

Datasets Metadata Algorithm Architecture 93

Netflix Tech

SEPTEMBER 18, 2024

message Item ( Bytes key, Bytes value, Metadata metadata, Integer chunk ) Database Agnostic Abstraction The KV abstraction is designed to hide the implementation details of the underlying database, offering a consistent interface to application developers regardless of the optimal storage system for that use case. . "persistence_configuration":[

Bytes 104

Bytes Metadata Database Data 104

Cloudera

AUGUST 10, 2021

For example, writing a Spark dataset to Ozone or launching a DDL query in Hive that points to a location in Ozone. I’ve chosen those names because I’ll be using an easy method for generating and writing TPC-DS datasets, along with creating their corresponding Hive tables. Create a dataset from the customer table. With CDP 7.1.4

Hadoop 105

Hadoop Kafka Datasets Government 105

Cloudera

AUGUST 17, 2022

Sophisticated data practitioners and business analysts want access to new datasets that can help optimize their work and transform whole business functions. Traditionally, it all starts with onboarding and transforming the datasets and then building analytical models that create business value, which can take weeks or months.

Cloud Computing 116

Cloud Computing Cloud Storage Data Science Government 116

Christophe Blefari

JULY 13, 2024

Fast News ⚡️ End-to-end data lineage in AWS — AWS announced DataZone to bring lineage to your data assets , from the picture it can mixes datasets (?), It provides abstractions and tools for the translation of lakehouse table format metadata. I'm not sure I'm happy to see this on Atlassian blog.

Kafka 130

Kafka AWS Data Database 130

Cloudera

SEPTEMBER 9, 2022

Iceberg is a next-generation, cloud-native table format designed to be open and scalable to petabyte datasets. With innovations like hidden partitioning and metadata stored at the file level, Iceberg makes querying on very large data sets faster, while also making changes to data easier and safer.

Architecture 102

Architecture Metadata Machine Learning Unstructured Data 102

Cloudera

APRIL 1, 2024

Our cutting-edge Shared data experience (SDX) service provides a unified control plane for common security, governance and metadata management on all structured and unstructured data. The post Cloudera Named a Visionary in the Gartner MQ for Cloud DBMS appeared first on Cloudera Blog. Unlike software, ML models need continuous tuning.

Cloud 105

Cloud Unstructured Data Metadata Government 105

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. In this blog post, we will highlight the work done recently to improve the performance of Ozone Manager to scale to exabytes of data. The hardware specifications are included at the end of this blog.

Management 70

Management Metadata Datasets Architecture 70

Cloudera

MARCH 23, 2022

Therefore, alleviating the need to use different connectors, exotic and poorly maintained APIs, and other use-case specific workarounds to work with your datasets. . Iceberg is designed to be open and engine agnostic allowing datasets to be shared. 3: Open Performance.

Metadata 92

Metadata Data Architecture Machine Learning BI 92

ProjectPro

FEBRUARY 8, 2023

Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. For analyzing huge datasets, they want to employ familiar Python primitive types. billion by 2026?

AWS 98

AWS Scala Metadata Data Lake 98

Netflix Tech

MARCH 25, 2019

In the rest of this blog, we will a) touch on the complexity of Netflix cloud landscape, b) discuss lineage design goals, ingestion architecture and the corresponding data model, c) share the challenges we faced and the learnings we picked up along the way, and d) close it out with “what’s next” on this journey. push or pull.

Building 90

Building Metadata Transportation Data Ingestion 90

Data Engineering Podcast

AUGUST 5, 2019

Once a dataset has been located, how does Amundsen simplify the process of accessing that data for analysis or further processing? Links Amundsen Data Council Presentation Strata Presentation Blog Post Lyft Airflow Podcast.__init__ Links Amundsen Data Council Presentation Strata Presentation Blog Post Lyft Airflow Podcast.__init__

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Cloudera

OCTOBER 11, 2021

The platform converges data cataloging, data ingestion, data profiling, data tagging, data discovery, and data exploration into a unified platform, driven by metadata. Modak Nabu reliably curates datasets for any line of business and personas, from business analysts to data scientists. Knowledge Graphs for the Business.

Data Engineer 95

Data Engineer Data Engineering Cloud Engineering 95