phData: Data Engineering

NOVEMBER 1, 2024

Monitor and Adapt: Continuously assess the impact of GenAI on data governance practices and be prepared to adapt policies as technologies evolve. Data governance is the only way to ensure those requirements are met. Chief Technology Officer, Finance Industry For all the quotes, download the Trendbook today!

Government 52

Government Data Governance Finance Metadata 52

Data Engineering Podcast

JUNE 17, 2023

In this episode Satish Jayanthi explores the benefits of incorporating column-aware tooling in the data modeling process. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Precisely

JUNE 6, 2023

At Precisely’s Trust ’23 conference, Chief Operating Officer Eric Yau hosted an expert panel discussion on modern data architectures. The group kicked off the session by exchanging ideas about what it means to have a modern data architecture.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

phData: Data Engineering

NOVEMBER 8, 2024

Track data files within the table along with their column statistics. Open table formats enable efficient data management and retrieval by storing these files chronologically, with a history of DDL and DML actions and an index of data file locations. Log all Inserts, Updates, and Deletes (DML) applied to the table.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Precisely

JULY 18, 2024

Key Takeaways Data Fabric is a modern data architecture that facilitates seamless data access, sharing, and management across an organization. Data management recommendations and data products emerge dynamically from the fabric through automation, activation, and AI/ML analysis of metadata.

Data Management 52

Data Management Management Metadata Database-centric 52

Cloudera

JUNE 7, 2022

Companies can now capitalize on the value in all their data, by delivering a hybrid data platform for modern data architectures with data anywhere. Cloudera Data Platform (CDP) is designed to address the critical requirements for modern data architectures today and tomorrow.

IT 112

IT Unstructured Data Data Architecture Government 112

Cloudera

OCTOBER 7, 2021

The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., need to integrate multiple “point solutions” used in a data ecosystem) and organization reasons (e.g., Components of a Data Mesh.

Architecture 124

Architecture Metadata Kafka Government 124

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Architecture 100

Architecture Metadata MongoDB Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 23, 2018

He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. How do you define data curation? How does the size and maturity of a company affect the ways that they architect and interact with their data systems?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

MARCH 23, 2022

In fact, we recently announced the integration with our cloud ecosystem bringing the benefits of Iceberg to enterprises as they make their journey to the public cloud, and as they adopt more converged architectures like the Lakehouse. 1: Multi-function analytics . 3: Open Performance.

Metadata 92

Metadata Data Architecture BI Machine Learning 92

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Precisely

DECEMBER 9, 2024

To name a few: privacy and security considerations compliance demands interest in emerging data management architectures like data mesh and data fabric increased AI adoption The findings show that data governance is the most-cited data challenge inhibiting progress toward AI initiatives (62%).

Data Governance 64

Data Governance Government Data Integration Architecture 64

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. Sample of a high-level data architecture blueprint for Azure BI programs.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

JULY 28, 2019

Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Data Integration 100

Data Integration Metadata Architecture Media 100

Data Engineering Podcast

AUGUST 5, 2019

Announcements Welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Data Engineering Podcast

SEPTEMBER 1, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Cloud AWS SQL 100

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 88

Data Lake Business Intelligence Metadata Data Warehouse 88

Data Engineering Podcast

OCTOBER 28, 2019

In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming. If you are building dataflows then Dagster is definitely worth exploring.

Building 100

Building Data Pipeline Programming Language Kafka 100

Data Engineering Podcast

OCTOBER 14, 2019

Dataform is a platform that helps you apply engineering principles to your data transformations and table definitions, including unit testing SQL scripts, defining repeatable pipelines, and adding metadata to your warehouse to improve your team’s communication. What do you have planned for the future of DataForm?

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Cloudera

OCTOBER 11, 2022

Different data types need different types of analytics – real-time, streaming, operational, data warehouses. As Mason said, all the data management, data analytics, and data science tools should easily work together and run against all this shared data. It should run on any cloud or on-prem.

Data Science 94

Data Science Aggregated Data Data Consulting 94

Towards Data Science

OCTOBER 25, 2023

Not too long ago, almost all data architectures and data team structures followed a centralized approach. As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. Your organization may be undergoing the decentralization of data.

Data Pipeline 98

Data Pipeline SQL Data Architecture Data 98

Data Engineering Weekly

DECEMBER 10, 2023

Capital One: Insights on building a data strategy to drive business value One of the hotly debated and many companies struggling with is to build an agile data strategy to drive business value. There is no shortage of data, but data management is still hard, which requires skillful execution and alignment across the org.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Towards Data Science

FEBRUARY 15, 2023

Data by itself has no value, it needs to be organized, standardized, and clean. In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture.

Data Lake 98

Data Lake Data Warehouse Hadoop Architecture 98

Cloudera

NOVEMBER 17, 2022

While Cloudera CDH was already a success story at HBL, in 2022, HBL identified the need to move its customer data centre environment from Cloudera’s CDH to Cloudera Data Platform (CDP) Private Cloud to accommodate growing volumes of data. See other customers’ success here .

Banking 85

Banking Management Data Lake Professional Services 85

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Databand.ai

JULY 19, 2023

This capability is useful for businesses, as it provides a clear and comprehensive view of their data’s history and transformations. Data lineage tools are not a new concept. In this article: Why Are Data Lineage Tools Important? It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Ascend.io

APRIL 19, 2023

In this post, I’ll explore the origins of the modern data stack, discuss why its promised benefits have proven elusive, and advocate for a post-modern approach to data management that prioritizes productivity and value. Where did the modern data stack come from?

Metadata 52

Metadata Hadoop Business Analyst Software Engineer 52

Cloudera

MAY 22, 2019

Cloudera’s data superheroes design modern data architectures that work across hybrid and multi-cloud and solve complex data management and analytic use cases spanning from the Edge to AI. DATA SECURITY AND GOVERNANCE.

Pharmaceutical 56

Pharmaceutical Recruitment Machine Learning Data Architecture 56

AltexSoft

NOVEMBER 10, 2021

Well, there’s a new phenomenon in data management that received the name of a data lakehouse. The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. Data warehouse.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. 2023 predictions from the panel are; Unified metadata becomes kingmaker.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

FEBRUARY 13, 2025

Summit Essentials Date & Location The Gartner Data & AI Summit takes place May 12-15th, 2025 in London, England. This year, the event will uncover the latest in data management, data trends, governance, and data architecture to deliver value for the future.

Data Analytics 52

Data Analytics Government Data Architecture Data Lake 52

Monte Carlo

MAY 25, 2023

That’s why we’re excited to announce that uniting diverse data team personas to collectively ensure data quality just got easier, thanks to a new integration between Monte Carlo’s data observability platform and Atlan’s active metadata platform.

Metadata 52

Metadata Data Governance Government Business Intelligence 52

Cloudera

JANUARY 5, 2023

In the past year, businesses who doubled down on digital transformation during the pandemic saw their efforts coming to fruition in the form of cost savings and more streamlined data management. These features provide businesses with a common metadata, security, and governance model across all their data.

Banking 52

Banking Machine Learning Insurance Data Architecture 52