dbt Developer Hub

APRIL 20, 2025

These tools can be called by LLM systems to learn about your data and metadata. Remember, as with any AI workflows, to make sure that you are taking appropriate caution in terms of giving these access to production systems and data. What is the best workflow for the current iteration of the MCP server?

Structured Data 98

Structured Data SQL BI Project 98

Ascend.io

DECEMBER 4, 2024

Deploy DataOps DataOps , or Data Operations, is an approach that applies the principles of DevOps to data management. It aims to streamline and automate data workflows, enhance collaboration and improve the agility of data teams. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Missing data? Missing data?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

JULY 17, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Weekly

NOVEMBER 3, 2024

Data Engineering Weekly readers get 15% discount by registering the following link, [link] Gustavo Akashi: Building data pipelines effortlessly with a DAG Builder for Apache Airflow Every code-first data workflow grew into a UI-based or Yaml-based workflow.

Data Engineer 63

Data Engineer Data Engineering Engineering Pipeline-centric 63

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. Linked data technologies provide a means of tightly coupling metadata with raw information.

Building 189

Building SQL BI Python 189

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Engineer 100

Data Engineer Data Engineering MongoDB Metadata 100

Engineering at Meta

FEBRUARY 4, 2025

For each data logs table, we initiate a new worker task that fetches the relevant metadata describing how to correctly query the data. Once we know what to query for a specific table, we create a task for each partition that executes a job in Dataswarm (our data pipeline system).

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. The only thing worse than having bad data is not knowing that you have it. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What are the contributing factors that lead to fragmentation of visibility for data workflows at different stages?

Building 100

Building Metadata MongoDB MySQL 100

Data Engineering Podcast

NOVEMBER 26, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 101

Metadata Data Pipeline Scala Data Process 101

Data Engineering Podcast

AUGUST 5, 2019

Finding the data that you need is tricky, and Amundsen will help you solve that problem. And as your data grows in volume and complexity, there are foundational principles that you can follow to keep data workflows streamlined. Finding the data that you need is tricky, and Amundsen will help you solve that problem.

MongoDB 100

MongoDB PostgreSQL Metadata Media 100

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 63

Metadata Data Warehouse ETL Tools Data Workflow 63

Data Engineering Podcast

JANUARY 30, 2022

You can observe your pipelines with built in metadata search and column level lineage. Finally, if you have existing workflows in AbInitio, Informatica or other ETL formats that you want to move to the cloud, you can import them automatically into Prophecy making them run productively on Spark.

Data Engineer 100

Data Engineer Data Engineering Engineering Python 100

Data Engineering Podcast

APRIL 5, 2021

What portions of the data workflow is Atlan responsible for? What components of the data stack might Atlan replace? How would you characterize Atlan’s position in the current data ecosystem? What makes Atlan stand out from other systems for data cataloguing, metadata management, or data governance?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

AltexSoft

JUNE 7, 2021

A HDFS Master Node, called a NameNode , keeps metadata with critical information about system files (like their names, locations, number of data blocks in the file, etc.) and keeps track of storage capacity, a volume of data being transferred, etc. Among solutions facilitation data management are. Apache Hadoop ecosystem.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Data Engineering Weekly

DECEMBER 16, 2024

Grab’s Metasense , Uber’s DataK9 , and Meta’s classification systems use AI to automatically categorize vast data sets, reducing manual efforts and improving accuracy. Beyond classification, organizations now use AI for automated metadata generation and data lineage tracking, creating more intelligent data infrastructures.

Data Engineer 63

Data Engineer Data Engineering Engineering Unstructured Data 63

Workfall

JUNE 12, 2023

It facilitates data synchronisation, replication, real-time analytics, and event-driven processing, empowering data-driven decision-making and operational efficiency. These additional columns store metadata like timestamps, user IDs, and change types, ensuring granular change tracking and auditability.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

RandomTrees

SEPTEMBER 17, 2024

The Unity Catalog is Databricks governance solution which integrates with Databricks workspaces and provides a centralized platform for managing metadata, data access, and security. Improved Data Discovery The tagging and documentation features in Unity Catalog facilitate better data discovery.

Data Governance 52

Data Governance Government Metadata Machine Learning 52

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Netflix Tech

OCTOBER 27, 2020

Netflix Scheduler is built on top of Meson which is a general purpose workflow orchestration and scheduling framework to execute and manage the lifecycle of the data workflow. Bulldozer makes data warehouse tables more accessible to different microservices and reduces each individual team’s burden to build their own solutions.

Data Warehouse 100

Data Warehouse Datasets Data Big Data 100

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. 2023 predictions from the panel are; Unified metadata becomes kingmaker. The author walked through various strategies, from sync to async job submission and batch job submission strategy.

Data Engineer 52

Data Engineer Data Engineering Engineering Metadata 52

dbt Developer Hub

OCTOBER 14, 2021

It uses the dbt Cloud Metadata API to surface metadata from dbt right in Hex, letting you quickly get the context you need on things like data freshness without juggling multiple apps and browser tabs. Hex just launched an integration with dbt! Get started here. Things to Watch ? What's missing? Spreadsheets?

Metadata 52

Metadata BI Software Engineer Software Engineering 52

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Ingestion 52

Netflix Tech

NOVEMBER 20, 2023

This enables auto propagation of backfill data in multi-stage pipelines. Netflix Maestro Maestro is the Netflix data workflow orchestration platform built to meet the current and future needs of Netflix. As we know, an iceberg table contains a list of snapshots with a set of metadata data.

Process 87

Process Data Pipeline Datasets SQL 87

Cloudera

FEBRUARY 11, 2019

Cloudera provides a unified platform with multiple data apps and tools, big data management, hybrid cloud deployment flexibility, admin tools for platform provisioning and control, and a shared data experience for centralized security, governance, and metadata management. 3. Expansion beyond core data management.

Data Management 43

Data Management Management Metadata Government 43

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. So, why is data orchestration a big deal? It automates and optimizes data processes, reducing manual effort and the likelihood of errors.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Databand.ai

AUGUST 30, 2023

DataOps tools should provide a comprehensive data cataloging solution that allows organizations to create a centralized repository of their data assets, complete with metadata, data lineage information, and data samples.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Weekly

MARCH 11, 2023

[link] Data Engineering Weekly Data Catalog - A Broken Promise Data catalogs are the most expensive data integration systems you never intended to build.

Data Engineer 52

Data Engineer Data Engineering Engineering High Quality Data 52

Netflix Tech

OCTOBER 18, 2022

With the high growth of workflows in the past few years?—?increasing increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. A workflow instance is an execution of a workflow, similarly, an execution of a step is called a step instance.

Java 84

Java Data Machine Learning Systems 84

Monte Carlo

OCTOBER 1, 2024

AI-powered Monitor Recommendations that leverage the power of data profiling to suggest appropriate monitors based on rich metadata and historic patterns — greatly simplifying the process of discovering, defining, and deploying field-specific monitors.

Database 52

Database Metadata Software Engineer Software Engineering 52

Hepta Analytics

FEBRUARY 14, 2022

Disadvantages of a data lake are: Can easily become a data swamp data has no versioning Same data with incompatible schemas is a problem without versioning Has no metadata associated It is difficult to join the data Data warehouse stores processed data, mostly structured data.

Data Ingestion 52

Data Ingestion Data Engineer Data Engineering Engineering 52

Cloudera

AUGUST 21, 2020

The governance aspect is perhaps even more important and businesses need to be able to understand where the data comes from. Data lineage, personally identifiable information or PPI and metadata all fall under a broad data governance banner which is critically important in terms of what needs to be protected and mapped out.

Banking 62

Banking Kafka Cloud Storage Government 62