Cloudera

MAY 10, 2024

Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time.

Government 103

Government Data Governance Data Pipeline Data Architecture 103

ProjectPro

JUNE 6, 2025

You must also learn about the best practices for efficient data loading, including optimizing for performance, handling errors, and automating data pipelines. By mastering data loading techniques, you can fill your Snowflake database with relevant data for further exploration and analysis in your Snowflake environment.

Data Warehouse 40

Data Warehouse SQL AWS Big Data 40

ProjectPro

JUNE 6, 2025

Snowflake SnowPro Advanced: Architect Certification Image Source: learn.snowflake.com/ This certification validates proficiency in implementing comprehensive architectural solutions using Snowflake. It covers data modeling, performance optimization, security, access control, and designing scalable data pipelines.

Certification 40

Certification Data Pipeline Hadoop AWS 40

Cloudera

FEBRUARY 15, 2024

Seeing the future in a modern data architecture The key to successfully navigating these challenges lies in the adoption of a modern data architecture. The promise of a modern data architecture might seem like a distant reality, but we at Cloudera believe data can make what is impossible today, possible tomorrow.

Insurance 88

Insurance Banking Data Architecture Data Ingestion 88

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

ProjectPro

JUNE 6, 2025

Traditional data preparation platforms, including Apache Spark, are unnecessarily complex and inefficient, resulting in fragile and costly data pipelines. Several organizations can quickly transform, integrate, and analyze their data with Snowflake's Data Cloud.

Architecture 56

Architecture IT Data Warehouse Amazon Web Services 56

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

ProjectPro

JUNE 6, 2025

It's designed to handle and analyze large volumes of data efficiently, making it a favorite choice for businesses of all sizes. Snowflake Features Multi-cluster, Shared Data Architecture: Snowflake's architecture is built around the concept of decoupling storage and compute.

Data Storage 58

Data Storage PostgreSQL Data Warehouse AWS 58

ProjectPro

JUNE 6, 2025

Kafka's popularity is skyrocketing, and for good reason—it helps organizations manage real-time data streams and build scalable data architectures. Learning how to connect Kafka with databases, Hadoop, Spark, or Flink will expand your knowledge of how Kafka is used in complex data pipelines.

Kafka 40

Kafka Certification AWS Retail 40

ProjectPro

JUNE 6, 2025

The result was Apache Iceberg, a modern table format built to handle the scale, performance, and flexibility demands of today’s cloud-native data architectures. Apache Iceberg Architecture 1. Data Layer What are the main use cases for Apache Iceberg? Ensure strong data governance and auditability.

Architecture 40

Architecture Data Lake Metadata Cloud Storage 40

ProjectPro

JUNE 6, 2025

This beginner's guide will give you a detailed overview of Azure Synapse Analytics and its architecture to help you build enterprise-grade data pipelines for your next data analytics project. Why Use Azure Synapse Analytics For Big Data Analytics Projects? Well, Azure Synapse Analytics is your answer!

Data Engineering 40

Data Engineering Data Engineer Engineering Data Lake 40

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Architecture 124

Architecture Metadata Kafka Government 124

Cloudera

NOVEMBER 18, 2020

The technological linchpin of its digital transformation has been its Enterprise Data Architecture & Governance platform. It hosts over 150 big data analytics sandboxes across the region with over 200 users utilizing the sandbox for data discovery.

Medical 123

Medical Banking Telecommunication Government 123

Cloudera

OCTOBER 19, 2023

While navigating so many simultaneous data-dependent transformations, they must balance the need to level up their data management practices—accelerating the rate at which they ingest, manage, prepare, and analyze data—with that of governing this data.

Telecommunication 76

Telecommunication Data Architecture Government Architecture 76

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is the main difference between a data architect and a data engineer?

Data Architect 87

Data Architect Certification Generalist Programming Language 87

ProjectPro

JUNE 6, 2025

It is based on the Hortonworks Data Platform 2.4 and covers five main categories: Data Access, Data Management, Data Governance, Workflow, Operations, and Security. Acquiring the HCA certification helps individuals showcase their knowledge in these areas and their ability to work with the Hortonworks Data Platform.

Big Data 40

Big Data Certification Amazon Web Services Hadoop 40

Cloudera

FEBRUARY 22, 2021

The Cloudera Data Platform (CDP) represents a paradigm shift in modern data architecture by addressing all existing and future analytical needs. Cloudera Data Catalog (part of SDX) replaces data governance tools to facilitate centralized data governance (data cataloging, data searching / lineage, tracking of data issues etc. ).

Data Warehouse 82

Data Warehouse Data Governance Government Cloud 82

Snowflake

DECEMBER 12, 2023

Iceberg Tables bring the easy management and great performance of Snowflake to data stored externally in an open source format. Snowflake Horizon Seamless data governance with a new user experience in Snowsight – general availability Users can effortlessly explore objects of interest and perform essential actions—all without SQL.

Metadata 118

Metadata Python Government AWS 118

Cloudera

DECEMBER 1, 2022

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. RI is a global leader in the design and deployment of large-scale, production-level modern data platforms for the world’s largest enterprises.

Data Lake 84

Data Lake Data Pipeline Data Governance Analytics Application 84

Databand.ai

AUGUST 30, 2023

It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. These systems typically consist of siloed data storage and processing environments, with manual processes and limited collaboration between teams.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Databand.ai

JULY 19, 2023

This capability is useful for businesses, as it provides a clear and comprehensive view of their data’s history and transformations. Data lineage tools are not a new concept. In this article: Why Are Data Lineage Tools Important? Atlan Atlan offers a modern approach to data governance.

Pipeline-centric 52

Pipeline-centric Data Governance Government Metadata 52

Cloudera

MARCH 23, 2022

In fact, we recently announced the integration with our cloud ecosystem bringing the benefits of Iceberg to enterprises as they make their journey to the public cloud, and as they adopt more converged architectures like the Lakehouse. Enterprise grade security and data governance – centralized data authorization to lineage and auditing.

Metadata 92

Metadata Data Architecture BI Machine Learning 92

ProjectPro

JUNE 6, 2025

Recommended Reading: Data Analyst Salary 2022-Based on Different Factors Data Engineer Data engineers are responsible for developing, constructing, and managing data pipelines. Creating dashboards and tools for business users based on analysis by data analysts and data scientists.

Data Science 56

Data Science Data Mining Data Architect BI 56

ProjectPro

JUNE 6, 2025

Examples of data products include tools or applications that process data and generate insights to aid in decision-making. The concept of data products is integral to the data mesh framework, emphasizing decentralized, domain-oriented data ownership, self-service platforms, and federated data governance.

Pipeline-centric 40

Pipeline-centric Database-centric Raw Data Data Warehouse 40

Data Engineering Weekly

MARCH 3, 2024

link] Zendesk: dbt at Zendesk The Zendesk team shares their journey of migrating legacy data pipelines to dbt, focusing on making them more reliable, efficient, and scalable. The article also highlights sink-specific improvements and operator-specific enhancements that contribute to the overall performance boost.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

ProjectPro

JUNE 6, 2025

Build a unique job-winning data engineer resume with big data mini projects. The professionals whose everyday duties involve gathering, transforming, and distributing data for data-driven decision-making should obtain this professional certificate as it will help them stand out from the competitors.

Certification 40

Certification Data Engineering Data Engineer Engineering 40

ProjectPro

JUNE 6, 2025

Kickstart your data engineer career with end-to-end solved big data projects for beginners. What does a Data Modeler do? The data modeler builds, implements, and analyzes data architecture and data modeling solutions using relational, dimensional, and NoSQL databases.

NoSQL 40

NoSQL ETL Tools SQL Certification 40

Ascend.io

OCTOBER 8, 2024

Data engineering is the backbone of any data-driven organization, responsible for building and maintaining the infrastructure that supports data collection, storage, and analysis. Traditionally, data engineers have focused on the technical aspects of data management, ensuring data pipelines run smoothly and efficiently.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Architecture 52

The Modern Data Company

JUNE 26, 2023

Strong data governance is essential Risks and challenges associated with using customer data–such as concerns around privacy and security–can derail insights. ABAC governance enables flexible and scalable policies that adapt to changing or new compliance regulations.

Retail 52

Retail Government Data Governance Unstructured Data 52

The Modern Data Company

JUNE 26, 2023

Strong data governance is essential Risks and challenges associated with using customer data–such as concerns around privacy and security–can derail insights. ABAC governance enables flexible and scalable policies that adapt to changing or new compliance regulations.

Retail 52

Retail Government Data Governance Unstructured Data 52

The Modern Data Company

JUNE 26, 2023

Strong data governance is essential Risks and challenges associated with using customer data–such as concerns around privacy and security–can derail insights. ABAC governance enables flexible and scalable policies that adapt to changing or new compliance regulations.

Retail 52

Retail Government Data Governance Unstructured Data 52

Knowledge Hut

MARCH 28, 2024

Job Role 1: Azure Data Engineer Azure Data Engineers develop, deploy, and manage data solutions with Microsoft Azure data services. They use many data storage, computation, and analytics technologies to develop scalable and robust data pipelines. GDPR, HIPAA), and industry standards.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

MARCH 13, 2024

Let’s dig in and explore the landscape of the top so-called “data quality tools” — what they are, what they’re not, and whether they’re the right first step towards more reliable data. Governance helps companies set important standards and achieve higher levels of data security, data accessibility, and data quality.

Data Governance 52

Data Governance Government Metadata Data 52

Knowledge Hut

MARCH 20, 2024

The job description for Azure data engineer that I have elucidated below focuses more on foundational tasks while providing opportunities for learning and growth within the field: Data ingestion: This role involves assisting in the process of collecting and importing data from various sources into Azure storage solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Certification 52

Knowledge Hut

MARCH 15, 2024

The job description for Azure data engineer that I have elucidated below focuses more on foundational tasks while providing opportunities for learning and growth within the field: Data ingestion: This role involves assisting in the process of collecting and importing data from various sources into Azure storage solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Certification 52

Data Engineering Weekly

FEBRUARY 25, 2024

link] PayPal: Leveraging Spark 3 and NVIDIA’s GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines PayPal's integration of Apache Spark 3 and NVIDIA GPUs has led to up to 70% cost savings in cloud expenses for big data and AI applications, processing petabytes of data across hundreds of thousands of jobs.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. The new kids on the table format block Within the last five or six years, as data management needs have grown in complexity and scale, newer table formats have emerged.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52