Data Engineering Podcast

JUNE 12, 2022

In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team. Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 130

Data Lake Data Ingestion MongoDB Google Cloud 130

Data Engineering Podcast

MAY 18, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

The Modern Data Company

APRIL 15, 2024

Understands how different pieces of data (books) relate to each other, helping users find related information. Keeps the library orderly, adjusting as new data arrives. Tracks key performance indicators to understand the use and value of the data. Manages information about data access and monitors data quality.

Government 52

Government Metadata Data Data Management 52

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Christophe Blefari

JUNE 16, 2023

The power of pre-commit and SQLFluff —SQL is a query programming language used to retrieve information from data storages, and like any other programming language, you need to enforce checks at all times. Malloy's Near Term Roadmap — I've shared recently Malloy demo , which was awesome. but I missed it).

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Monte Carlo

MAY 20, 2024

On the other hand, a data catalog aims to provide a macro view of an organization’s data landscape. It emphasizes discoverability, accessibility, and governance of data assets, promoting an integrated approach to data management across multiple platforms and stakeholders.

Datasets 52

Datasets Data Government Business Analyst 52

Monte Carlo

MAY 25, 2023

That’s why we’re excited to announce that uniting diverse data team personas to collectively ensure data quality just got easier, thanks to a new integration between Monte Carlo’s data observability platform and Atlan’s active metadata platform.

Metadata 52

Metadata Data Governance Government Business Intelligence 52

Data Engineering Podcast

SEPTEMBER 23, 2018

He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Precisely

MAY 31, 2023

Users cannot find the data they need, gain access in a timely fashion, rely on data being in a useable format, or understand its impact on the business – meaning that it is unusable to them. The end result is that users can’t trust their data because it lacks integrity. In many cases, that means real-time availability.

Data Integration 52

Data Integration Metadata Data Governance Data 52

The Modern Data Company

MARCH 7, 2023

A data taxonomy uses a system of unambiguous metadata terms (such as a filename or tags attached to a file) that allow an enterprise to classify a file or dataset into important business categories. How Does Taxonomy Aid Data Governance? Incompleteness – Data may be missing important attributes.

Data Governance 52

Data Governance Government Metadata Datasets 52

The Modern Data Company

JANUARY 17, 2023

These include internal data lakes, apps (1st or 3rd party), web SDK, internal or 3rd-party databases, and even spreadsheets generated within the marketing department. Without lineage or governance metadata, the quality of this data is anyone’s guess. New data is added to the catalog automatically upon ingestion.

Metadata 59

Metadata Government Data Lake Data 59

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. AWS is one of the most popular data lake vendors.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Ascend.io

MAY 23, 2023

Orphaned datasets occur due to several reasons, such as changes in data relationships, ad-hoc archiving of deleted records, restarts from system failures and errors, or design flaws in data management strategies. This strategy minimizes clutter and storage space, but could lose some data that is no longer useful going forward.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Cloudera

AUGUST 31, 2021

Running on CDW is fully integrated with streaming, data engineering, and machine learning analytics. It has a consistent framework that secures and provides governance for all data and metadata on private clouds, multiple public clouds, or hybrid clouds. Consideration of both data & metadata in the migration.

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

phData: Data Engineering

DECEMBER 16, 2021

As a result, it is important to know what is happening to your data. Is your data just being scanned, or is it being moved? Where is your metadata stored? How easy is it to offboard your data if you choose another tool? It can let the user know about the quality and age of the data as well as any policies attached to it.

Metadata 52

Metadata Datasets Cloud ETL Tools 52

AltexSoft

MARCH 2, 2023

However, decentralized models may result in inconsistent and duplicate master data. There’s a centralized structure that provides a framework, which is then used by autonomous departments that own their data and metadata. Data management is the overall process of collecting, storing, organizing, maintaining, and using data.

Data Governance 52

Data Governance Government Programming Healthcare 52

Cloudera

JUNE 25, 2019

Cloudera shared a comprehensive overview and demonstration of the all-new Cloudera Data Platform (CDP). This industry-first enterprise data cloud offering will include cloud-native services for data warehousing, machine learning, streaming ingest, and operational data stores.

Cloud 86

Cloud Entertainment Government Machine Learning 86

Cloudera

MARCH 15, 2023

In this post I am going to set up a demo environment with a Spring Boot microservice and a streaming cluster using Cloudera Public Cloud. The Outbox Pattern The general idea behind this pattern is to have an “outbox” table in the service’s data store.

PostgreSQL 89

PostgreSQL Kafka Database Data 89

AltexSoft

OCTOBER 8, 2021

The bad news is, integrating data can become a tedious task, especially when done manually. Luckily, there are various data integration tools that support automation and provide a unified data view for more efficient data management. Data integration process. Data profiling and cleansing.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Data Engineering Podcast

JANUARY 28, 2018

Your host is Tobias Macey and today I’m interviewing Danielle Robinson and Joe Hand about Dat Project, a distributed data sharing protocol for building applications of the future Interview Introduction How did you get involved in the area of data management? What is the Dat project and how did it get started?

Data 100

Data Project Data Management Electronics 100

The Modern Data Company

JANUARY 22, 2024

The Modern Story: Navigating Complexity and Rethinking Data in The Business Landscape Enterprises face a data landscape marked by the proliferation of IoT-generated data, an influx of unstructured data, and a pervasive need for comprehensive data analytics.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

The Modern Data Company

JANUARY 22, 2024

The Modern Story: Navigating Complexity and Rethinking Data in The Business Landscape Enterprises face a data landscape marked by the proliferation of IoT-generated data, an influx of unstructured data, and a pervasive need for comprehensive data analytics.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Precisely

JULY 15, 2024

Key Takeaways Data Mesh is a modern data management architectural strategy that decentralizes development of trusted data products to support real-time business decisions and analytics. However, complex architectures and data silos make that difficult. One strategy being leveraged is a data mesh.

Data Governance 72

Data Governance Data Integration Government Data 72

The Modern Data Company

MARCH 1, 2024

Data Products work similarly, offering pre-packaged, actionable data units tailored to specific business goals. We will delve into the world of Data Products, exploring their transformative potential for CDOs and data teams. Data Products simplify access and processing, empowering faster decision-making.

Database-centric 40

Database-centric Data Lake Government Retail 40

Data Engineering Weekly

JULY 23, 2023

link] Piethein Strengholt: Data Management at Scale Data Mesh is a widely discussed and debated topic on the possibility of decentralized data management at scale. The author unpacks the federated design for data management in a phase-by-phase manner by adding more components as expanding.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Monte Carlo

APRIL 20, 2023

I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen. This way no decisions get made on bad data and our team becomes a proactive part of the solution,” said then Senior Director of Data at Freshly, Vitaly Lilich.

Data Warehouse 52

Data Warehouse BI Data Government 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64