Cloudera

NOVEMBER 15, 2024

By leveraging Apache Iceberg, an open table format designed for high-performance analytics on massive volumes of data, cyber defenders can access all of their data and conduct investigations with greater speed and efficiency. Whether they need to query data from today or from years past, the system scales up or down to meet their needs.

Metadata 76

Metadata Unstructured Data Data Lake Government 76

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Data Engineering Weekly

JUNE 16, 2024

The blog explains KIP-932 and its potential benefits. link] Yousry Mohamed: Delta Lake Liquid Clustering — A visual explanation Liquid clustering liberates the hive-style static partitioning and organizes the data layout from the accessing pattern. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease!

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Metadata 109

Metadata Government Data Governance Data 109

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 103

Government Accessibility Accessible Cloud 103

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 99

Bytes Metadata Database Data 99

Pinterest Engineering

OCTOBER 2, 2024

Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc., and also providing runtime metadata about the platform Security and Auditability , which ensures GDPR compliance, advanced levels of auditability, security, AAA, etc.

Data Management 56

Data Management Management Metadata SQL 56

Cloudera

NOVEMBER 15, 2024

By leveraging Apache Iceberg, an open table format designed for high-performance analytics on massive volumes of data, cyber defenders can access all of their data and conduct investigations with greater speed and efficiency. The post Empower Your Cyber Defenders with Real-Time Analytics appeared first on Cloudera Blog.

Metadata 52

Metadata Unstructured Data Data Lake Government 52

phData: Data Engineering

NOVEMBER 8, 2024

In this blog, we will discuss: What is the Open Table format (OTF)? Then, we add another column called HASHKEY , add more data, and locate the S3 file containing metadata for the iceberg table. In the screenshot below, we can see that the metadata file for the Iceberg table retains the snapshot history. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Weekly

JUNE 24, 2024

link] Sponsored: 2024 State of Apache Airflow Report Gain access to the latest trends and insights shaping the world of Apache Airflow—the go-to platform for data pipeline development and orchestration. The blog contains a summary of each talk and a link to the YouTube channel with all the talks.

Data Engineer 109

Data Engineer Data Engineering Engineering Google Cloud 109

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. More details on the research behind the Cortex Search retrieval stack will be shared on our Snowflake Engineering Blog.

Unstructured Data 113

Unstructured Data Metadata Government SQL 113

Cloudera

APRIL 2, 2024

We understand the importance of protecting sensitive government data from unauthorized access, and we have implemented robust security measures to safeguard this information. The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

Designing 100

Designing Process Government Metadata 100

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Bytes 94

Bytes Datasets Metadata Data 94

Cloudera

OCTOBER 10, 2024

Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance. The post Cloudera Lakehouse Optimizer Makes it Easier Than Ever to Deliver High-Performance Iceberg Tables appeared first on Cloudera Blog.

IT 83

IT Data Lake Metadata Data Warehouse 83

Precisely

AUGUST 22, 2024

Both architectures tackle significant data management challenges such as integrating disparate data sources, improving data accessibility, automating management processes, and ensuring data governance and security. Data fabric centralizes data governance and security to ensure privacy, secure access, and high-quality data.

Data Governance 69

Data Governance Government High Quality Data Metadata 69

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. These additional inputs are referred to as metadata filtering.

Cloud 64

Cloud Building Metadata Kafka 64

phData: Data Engineering

NOVEMBER 1, 2024

This blog is a collection of those insights, but for the full trendbook, we recommend downloading the PDF. It serves as a vital protective measure, ensuring proper data access while managing risks like data breaches and unauthorized use. of respondents rating it as highly impactful. They can identify where risks are and what to avoid.

Government 52

Government Data Governance Finance Metadata 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 99

Metadata Data Pipeline Scala Data Workflow 99

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. The data files and metadata files in Iceberg format are immutable.

Java 69

Java Metadata PostgreSQL Data Warehouse 69

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. link] Gunnar Morling: Revisiting the Outbox Pattern The blog is an excellent summary of the path we crossed with the outbox pattern and the challenges ahead. What are you waiting for?

Data Engineer 62

Data Engineer Data Engineering Engineering Pipeline-centric 62

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI.

Cloud 96

Cloud Kafka Metadata SQL 96

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 135

MySQL Metadata Data Validation Coding 135

Data Engineering Weekly

SEPTEMBER 15, 2024

link] Miro: Data Products Reliability - The Power of Metadata Miro writes about its adoption of Data Contracts. The blog narrates how the lack of neutral guidance or standards leads to ambiguity and the formation of a central team to bring trust in data. Though the blog talks specifically to Beam, every pipeline can adopt the theme.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. In this blog post, we will highlight the work done recently to improve the performance of Ozone Manager to scale to exabytes of data. The hardware specifications are included at the end of this blog.

Management 69

Management Metadata Datasets Architecture 69

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables (not files/blobs) are the only API abstraction for end-users.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 96

Management Kafka Metadata Media 96

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Metadata 81

Metadata Government Data Lake Data 81

Cloudera

DECEMBER 15, 2022

With CDP, customers can deploy storage, compute, and access, all with the freedom offered by the cloud, avoiding vendor lock-in and taking advantage of best-of-breed solutions. Only metadata will be regenerated. Only metadata will be regenerated. Amazingly fast table migration. Data quality using table rollback.

Cloud 78

Cloud Metadata Google Cloud Data Warehouse 78

Cloudera

SEPTEMBER 15, 2022

In this blog post, we will talk about a single Ozone cluster with the capabilities of both Hadoop Core File System (HCFS) and Object Store (like Amazon S3). Interoperability of the same data for several workloads: multi-protocol access. Provides high performance namespace metadata operations similar to HDFS. LEGACY Bucket.

Systems 86

Systems Hadoop Metadata Telecommunication 86

Databand.ai

AUGUST 30, 2023

Learn more in our detailed guide to data reliability testing Metadata Management Metadata management is the process of organizing, maintaining, and using metadata to improve the quality, consistency, and usability of your data.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 138

Algorithm Metadata Kafka Systems 138

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. A proper survey of these approaches would fill many blog posts of this size. The same applies to the metadata of your system. Be very wary of homerolled “vector-search infra.”

Metadata 52

Metadata Database Algorithm SQL 52

Cloudera

FEBRUARY 22, 2022

Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . In CDP we enable Iceberg tables side-by-side with the Hive table types, both of which are part of our SDX metadata and security framework.

Metadata 108

Metadata Datasets BI SQL 108

Data Engineering Weekly

AUGUST 25, 2024

The blog is a good overview of various components in a typical data stack. Powerful deep learning models are becoming smarter, more accessible and cost-effective. The blog narrates the shift-left approach in data governance with three critical principles. link] All rights reserved ProtoGrowth Inc, India.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70