Data Engineering Weekly

MARCH 5, 2025

This blog post expands on that insightful conversation, offering a critical look at Iceberg's potential and the hurdles organizations face when adopting it. This ecosystem includes: Catalogs: Services that manage metadata about Iceberg tables (e.g., If not handled correctly, managing this metadata can become a bottleneck.

Hadoop 58

Hadoop Metadata Data Ingestion Data Governance 58

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. This is what managing data without metadata feels like. This is what managing data without metadata feels like. Effective metadata management is no longer a luxury—it’s a necessity.

Metadata 52

Metadata IT Government High Quality Data 52

phData: Data Engineering

NOVEMBER 1, 2024

This blog is a collection of those insights, but for the full trendbook, we recommend downloading the PDF. It serves as a vital protective measure, ensuring proper data access while managing risks like data breaches and unauthorized use. of respondents rating it as highly impactful. They can identify where risks are and what to avoid.

Government 52

Government Data Governance Finance Metadata 52

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Unlike traditional planners that need to consider accessing a table via a variety of types of index, Impala’s planner always starts with a full table scan and then applies pruning techniques to reduce the data scanned. Metadata Caching. Execution Engine.

Metadata 143

Metadata Coding SQL Database 143

phData: Data Engineering

NOVEMBER 8, 2024

In this blog, we will discuss: What is the Open Table format (OTF)? Then, we add another column called HASHKEY , add more data, and locate the S3 file containing metadata for the iceberg table. In the screenshot below, we can see that the metadata file for the Iceberg table retains the snapshot history. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Cloudera

MARCH 4, 2024

In this article, we will walk you through the process of implementing fine grained access control for the data governance framework within the Cloudera platform. In a good data governance strategy, it is important to define roles that allow the business to limit the level of access that users can have to their strategic data assets.

Metadata 52

Metadata Data Governance Government Management 52

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Are your tools simple to implement and accessible to users with diverse skill sets? Have you considered whether your data platform allows easy access, integration, and management of data by different teams?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. link] Gunnar Morling: Revisiting the Outbox Pattern The blog is an excellent summary of the path we crossed with the outbox pattern and the challenges ahead. What are you waiting for?

Data Engineering 63

Data Engineering Data Engineer Engineering Pipeline-centric 63

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Data and Metadata: Data inputs and data outputs produced based on the application logic. Introduction.

Architecture 124

Architecture Metadata Kafka Government 124

Data Engineering Weekly

FEBRUARY 23, 2025

It covers nine categories: storage systems, data lake platforms, processing, integration, orchestration, infrastructure, ML/AI, metadata management, and analytics. I found the blog to be a comprehensive roadmap for data engineering in 2025.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Snowflake

APRIL 30, 2025

Ransomware is a type of malicious software that encrypts a victim's data or locks their device, demanding a ransom to restore access or to not expose the data. Prevention Access controls Access to Snowflake infrastructure is protected by multiple layers of security (defense in depth and zero trust ).

Accessible 52

Accessible Accessibility Metadata Cloud 52

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 116

Government Accessible Accessibility Cloud 116

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. More details on the research behind the Cortex Search retrieval stack will be shared on our Snowflake Engineering Blog.

Unstructured Data 132

Unstructured Data Metadata Government SQL 132

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Weekly

JUNE 16, 2024

The blog explains KIP-932 and its potential benefits. link] Yousry Mohamed: Delta Lake Liquid Clustering — A visual explanation Liquid clustering liberates the hive-style static partitioning and organizes the data layout from the accessing pattern. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease!

Data Engineering 105

Data Engineering Data Engineer Engineering Kafka 105

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 135

MySQL Metadata Coding Data Validation 135

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. This is particularly useful in environments where multiple applications need to access and process the same data. What is Change Data Capture? or its affiliates.

Kafka 50

Kafka MySQL Database Software Engineer 50

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Are your tools simple to implement and accessible to users with diverse skill sets? Have you considered whether your data platform allows easy access, integration, and management of data by different teams?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI.

Cloud 97

Cloud Kafka Metadata SQL 97

Data Engineering Weekly

MARCH 23, 2025

I honestly don’t have a solid answer, but this blog is an excellent overview of upskilling. [link] Georg Heiler: Upskilling data engineers What should I prefer for 2028, or how can I break into data engineering? These are common LinkedIn requests. and then to Nuage 3.0, The article highlights Nuage 3.0's

Data Engineering 63

Data Engineering Data Engineer Engineering Data 63

Precisely

DECEMBER 9, 2024

Fueled by businesses demand for democratized, self-service data, these architectures rely on effective metadata management and data governance for success. And, stay tuned for the next 2025 Planning Insights blog to explore more report highlights! It follows that 25% of respondents identify a data catalog as a top priority for 2024.

Data Governance 64

Data Governance Government Data Integration Architecture 64

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Picnic Engineering

FEBRUARY 6, 2025

In our previous blogs, we explored how Picnics Page Platform transformed the way we build new featuresenabling faster iteration, tighter collaboration and less feature-specific complexity. In this blog, well dive into how we configure the pages within Picnics store. Now, were bringing this same principle to our app.

Metadata 52

Metadata Architecture SQL Building 52

Cloudera

FEBRUARY 22, 2022

Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . In CDP we enable Iceberg tables side-by-side with the Hive table types, both of which are part of our SDX metadata and security framework.

Metadata 110

Metadata Datasets BI SQL 110

Netflix Tech

DECEMBER 3, 2022

All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command. This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Metadata 110

Metadata Government Data Governance Data 110

Data Engineering Weekly

JUNE 24, 2024

link] Sponsored: 2024 State of Apache Airflow Report Gain access to the latest trends and insights shaping the world of Apache Airflow—the go-to platform for data pipeline development and orchestration. The blog contains a summary of each talk and a link to the YouTube channel with all the talks.

Data Engineering 110

Data Engineering Data Engineer Engineering Google Cloud 110

Netflix Tech

FEBRUARY 14, 2025

In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. It filters out any invalid entries and enriches the valid ones with additional metadata, such as show or movie title details, and the specific page and row location where each impression was presented to users.

Kafka 66

Kafka Datasets Metadata Utilities 66

Cloudera

SEPTEMBER 9, 2022

Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. These new formats need to accommodate the massive scale increases while shortening the response windows for accessing, analyzing, and using these data sets for business insights.

Architecture 102

Architecture Metadata Machine Learning Unstructured Data 102

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. Find out more about unlocking the potential of your data in this whitepaper.

Data Governance 100

Data Governance Government Metadata Medical 100

Snowflake

NOVEMBER 2, 2023

To give customers flexibility for how they fit Snowflake into their architecture, Iceberg Tables can be configured to use either Snowflake or an external service like AWS Glue as the tables’s catalog to track metadata, with an easy one-line SQL command to convert to Snowflake in a metadata-only operation.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 104

Bytes Metadata Database Data 104

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

Process 114

Process SQL Kafka Database 114

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 105

Metadata AWS Data Lake Datasets 105

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Introduction and Rationale. Networking .

Architecture 102

Architecture Cloud Kafka Hadoop 102