Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Weekly

JUNE 16, 2024

The blog explains KIP-932 and its potential benefits. link] Yousry Mohamed: Delta Lake Liquid Clustering — A visual explanation Liquid clustering liberates the hive-style static partitioning and organizes the data layout from the accessing pattern. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease!

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 134

MySQL Metadata Data Validation Coding 134

Cloudera

SEPTEMBER 28, 2021

Cloudera Data platform ( CDP ) provides a Shared Data Experience ( SDX ) for centralized data access control and audit in the Enterprise Data Cloud. The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. Changes with file access control .

Accessibility 62

Accessibility Accessible Cloud Cloud Storage 62

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI.

Cloud 97

Cloud Kafka Metadata SQL 97

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Metadata Kafka Systems 139

Cloudera

FEBRUARY 22, 2022

Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . In CDP we enable Iceberg tables side-by-side with the Hive table types, both of which are part of our SDX metadata and security framework.

Metadata 111

Metadata Datasets BI SQL 111

Netflix Tech

DECEMBER 3, 2022

All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command. This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 102

Government Accessibility Accessible Cloud 102

Cloudera

APRIL 2, 2024

We understand the importance of protecting sensitive government data from unauthorized access, and we have implemented robust security measures to safeguard this information. The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

Designing 102

Designing Process Government Metadata 102

Data Engineering Weekly

JUNE 24, 2024

link] Sponsored: 2024 State of Apache Airflow Report Gain access to the latest trends and insights shaping the world of Apache Airflow—the go-to platform for data pipeline development and orchestration. The blog contains a summary of each talk and a link to the YouTube channel with all the talks.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Netflix Tech

SEPTEMBER 2, 2020

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. In an earlier blog post, we discussed Telltale , our health monitoring system. by Elizabeth Carretto Everyone loves Unsolved Mysteries. What is Edgar?

Metadata 119

Metadata Transportation Engineering Systems 119

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. More details on the research behind the Cortex Search retrieval stack will be shared on our Snowflake Engineering Blog.

Unstructured Data 112

Unstructured Data Metadata Government SQL 112

Cloudera

SEPTEMBER 9, 2022

Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. These new formats need to accommodate the massive scale increases while shortening the response windows for accessing, analyzing, and using these data sets for business insights.

Architecture 103

Architecture Metadata Machine Learning Unstructured Data 103

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. Find out more about unlocking the potential of your data in this whitepaper.

Data Governance 101

Data Governance Government Metadata Medical 101

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 104

Bytes Metadata Database Data 104

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

Process 114

Process SQL Kafka Database 114

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 119

Data Pipeline Building Manufacturing Data Warehouse 119

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Introduction and Rationale. Networking .

Architecture 103

Architecture Cloud Kafka Hadoop 103

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 97

Management Kafka Metadata Media 97

Cloudera

MAY 2, 2022

This is not just to implement specific governance rules — such as tagging, metadata management, access controls, or anonymization — but to prepare for the potential for rules to change in the future. . Governance must also work hand in hand with security to restrict who can access what data, or restrict where it is permitted to reside.

Cloud 105

Cloud Government Healthcare Metadata 105

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 101

Metadata Data Pipeline Scala Data Workflow 101

Cloudera

AUGUST 3, 2021

Access to innovation: The open source community drives change, innovation, and feature functionality at a higher rate than any one organization, even with heavy R&D investment. . Flexibility and choice: Organizations always have access to their data, and don’t get locked-in by any vendor. Keep data lineage secure and governed.

Manufacturing 111

Manufacturing Retail Government Entertainment 111

Cloudera

DECEMBER 19, 2024

However, they still need to provide access to a unified view of that data for many use casesfrom customer analytics to operational efficiency. Clouderas investment in and support for open metadata standards, our true hybrid architecture, and our native Spark offering for Iceberg combine to make us the ideal Iceberg data lakehouse.

AWS 75

AWS Metadata Government Machine Learning 75

Cloudera

SEPTEMBER 15, 2022

In this blog post, we will talk about a single Ozone cluster with the capabilities of both Hadoop Core File System (HCFS) and Object Store (like Amazon S3). Interoperability of the same data for several workloads: multi-protocol access. Provides high performance namespace metadata operations similar to HDFS. LEGACY Bucket.

Systems 88

Systems Hadoop Metadata Telecommunication 88

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Maxime Beauchemin

JANUARY 20, 2017

The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own. In retrospect, complex SCD modeling techniques are not intuitive and reduce accessibility.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Cloudera

DECEMBER 3, 2024

Cloudera’s open data lakehouse, powered by Apache Iceberg, solves the real-world big data challenges mentioned above by providing a unified, curated, shareable, and interoperable data lake that is accessible by a wide array of Iceberg-compatible compute engines and tools. Setup REST Catalog – link to REST Catalog Setup blog 5.

Metadata 85

Metadata SQL Database Data Warehouse 85

Cloudera

DECEMBER 10, 2020

In a multi-tenant environment, many users access the same data sources. The very high degree of compression built into these file formats reduces storage space and time to access storage. These techniques range from distributing concurrent query for overall throughput to metadata caching, data caching, and results caching.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Confluent

JULY 1, 2019

When a client (producer/consumer) starts, it will request metadata about which broker is the leader for a partition—and it can do this from any broker. The key thing is that when you run a client, the broker you pass to it is just where it’s going to go and get the metadata about brokers in the cluster from. The default is 0.0.0.0,

Kafka 101

Kafka Metadata AWS Bytes 101

Cloudera

SEPTEMBER 28, 2020

Attribute-based access control and SparkSQL fine-grained access control. Store and access schemas across clusters and rebalance clusters with Cruise Control. The post Upgrade Journey: The Path from CDH to CDP Private Cloud appeared first on Cloudera Blog. Gateway-based SSO with Knox. Ranger 2.0. on roadmap).

Cloud 132

Cloud Kafka Professional Services Metadata 132

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. NMDB is built to be a highly scalable, multi-tenant, media metadata system that can serve a high volume of write/read throughput as well as support near real-time queries.

Media 97

Media Database Metadata Data Schemas 97

Tweag

OCTOBER 25, 2023

Maintainers also add all sorts of metadata such as the name, the version, the license, the homepage, the description, and so on, which could be used to make an even richer graph. nixtract ’s goal is to make it accessible in a way that’s easier to consume. nixtract ’s goal is to make it accessible in a way that’s easier to consume.

Metadata 114

Metadata Accessibility Accessible Python 114

Maxime Beauchemin

AUGUST 28, 2017

Typically, the metadata around data lineage is usually incomplete or is buried in code that only a select few will have the capacity and patience to read. I’m planning an upcoming blog post titled “Next generation, data-aware ETL” where I’ll be proposing a design for a new framework that has accessibility and maintainability at its very core.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Cloudera

DECEMBER 20, 2021

But balancing a strong layer of security and governance with easy access to data for all users is no easy task. Such complex data calls for an advanced architecture, provided by Cloudera, that supports data & metadata management, analysis, security, and governance, and automates data pipelines & quality checks.

Government 92

Government Data Security Banking Metadata 92

Cloudera

NOVEMBER 15, 2024

By leveraging Apache Iceberg, an open table format designed for high-performance analytics on massive volumes of data, cyber defenders can access all of their data and conduct investigations with greater speed and efficiency. Whether they need to query data from today or from years past, the system scales up or down to meet their needs.

Metadata 85

Metadata Unstructured Data Data Lake Government 85

Netflix Tech

MARCH 25, 2019

In the rest of this blog, we will a) touch on the complexity of Netflix cloud landscape, b) discuss lineage design goals, ingestion architecture and the corresponding data model, c) share the challenges we faced and the learnings we picked up along the way, and d) close it out with “what’s next” on this journey. push or pull.

Building 90

Building Metadata Transportation Data Ingestion 90