Data Engineering Weekly

NOVEMBER 24, 2024

Canva writes about its custom solution using dbt and metadata capturing to attribute costs, monitor performance, and enable data-driven decision-making, significantly enhancing its Snowflake environment management. link] Grab: Metasense V2 - Enhancing, improving, and productionisation of LLM-powered data governance.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

François Nguyen

JANUARY 18, 2021

To finish the trilogy (Dataops, MLops), let’s talk about DataGovOps or how you can support your Data Governance initiative. In every step,we do not just read, transform and write data, we are also doing that with the metadata. Last part, it was added the data security and privacy part. What data do we have ?

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Ascend.io

DECEMBER 4, 2024

As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. The anticipated growth in data pipelines presents both challenges and opportunities.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. It enhances data quality, governance, and automation, transforming raw data into valuable insights. This is what managing data without metadata feels like.

Metadata 52

Metadata IT Government High Quality Data 52

Knowledge Hut

APRIL 20, 2023

Data governance refers to the set of policies, procedures, mix of people and standards that organisations put in place to manage their data assets. It involves establishing a framework for data management that ensures data quality, privacy, security, and compliance with regulatory requirements.

Data Governance 59

Data Governance Government Data Cleanse Data Security 59

The Modern Data Company

JANUARY 22, 2024

This mission culminates in the esteemed recognition of honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration, showcasing commitment to excellence and industry leadership in the data-driven era. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Data Science Blog: Data Engineering

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. They transform data into a consistent format for users to consume.

Data Pipeline 52

Data Pipeline BI Data Lake Data Warehouse 52

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Machine Learning 100

Data Engineering Podcast

NOVEMBER 26, 2023

Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

François Nguyen

MARCH 7, 2021

In this article, Juan Sequada gives maybe one of the best definition of Data Mesh ” It is paradigm shift towards a distributed architecture that attempts to find an ideal balance between centralization and decentralization of metadata and data management.”

Government 130

Government Data Governance Data Metadata 130

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 118

Metadata Python AWS Government 118

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB MySQL 100

Cloudera

OCTOBER 7, 2021

Application Logic: Application logic refers to the type of data processing, and can be anything from analytical or operational systems to data pipelines that ingest data inputs, apply transformations based on some business logic and produce data outputs.

Architecture 124

Architecture Metadata Kafka Government 124

The Modern Data Company

JANUARY 22, 2024

This mission culminates in the esteemed recognition of honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration, showcasing commitment to excellence and industry leadership in the data-driven era. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Cloudera

DECEMBER 20, 2021

Understanding that the future of banking is data-driven and cloud-based, Bank of the West embraced cloud computing and its benefits, like remote capabilities, integrated processes, and flexible systems. The platform is centralizing the data, data management & governance, and building custom controls for data ingestion into the system.

Government 91

Government Data Security Banking Metadata 91

Data Engineering Podcast

APRIL 5, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Data Engineering Podcast

NOVEMBER 9, 2020

In this episode Grant Seward explains how he built Tree Schema to be an easy to use and cost effective option for organizations to build their data catalogs. He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems.

Building 100

Building PostgreSQL BI Metadata 100

Databand.ai

JULY 19, 2023

However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. In this article: Why Are Data Lineage Tools Important? It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Cloudera

MARCH 23, 2022

Through Cloudera’s contributions, we have extended support for Hive and Impala, delivering on the vision of a data architecture for multi-function analytics from large scale data engineering (DE) workloads and stream processing (DF) to fast BI and querying (within DW) and machine learning (ML). . 3: Open Performance.

Metadata 92

Metadata Data Architecture BI Machine Learning 92

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. A notable capability that achieves this is the data catalog. If so, how do you combine that metadata with other data across the enterprise? #4.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

AltexSoft

FEBRUARY 11, 2023

Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is the main difference between a data architect and a data engineer? This privacy law must be kept in mind when building data architecture.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Precisely

JULY 15, 2024

Watch Preparing for a Data Mesh Strategy Key pillars when preparing for a data mesh strategy include: A mature data governance strategy to manage and organize a decentralized data system. Proper governance ensures that data is uniformly accessible and the appropriate security measures are met.

Data Governance 72

Data Governance Data Integration Government Data 72

Monte Carlo

MAY 18, 2023

Whether you’re a data scientist, data engineer, or business analyst, keeping track of your data’s origin, transformation, and movement is crucial for maintaining transparency, enforcing data governance, and ensuring data quality. Why do you need data lineage?

Metadata 64

Metadata Data Lake Government Data Governance 64

Data Engineering Weekly

DECEMBER 16, 2024

Grab’s Metasense , Uber’s DataK9 , and Meta’s classification systems use AI to automatically categorize vast data sets, reducing manual efforts and improving accuracy. Beyond classification, organizations now use AI for automated metadata generation and data lineage tracking, creating more intelligent data infrastructures.

Data Engineering 62

Data Engineering Data Engineer Engineering Unstructured Data 62

Precisely

MAY 23, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it is important to leverage tools that have the capabilities to support forward-looking use cases. A notable capability that achieves this is the data catalog. Deployment should be resource-efficient and easily targeted to fit your use cases.

Data Integration 64

Data Integration Metadata Telecommunication Data Governance 64

Ascend.io

APRIL 7, 2023

With the monolithic architectures most organizations have today, business users are stuck, constantly waiting for new data pipelines to be built or amended based on their requests. Data engineers aren’t huge fans of this paradigm either. Anyone can query the metadata any time anywhere to get the information they need.

Metadata 52

Metadata Data Governance Datasets Government 52

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Cloudera

NOVEMBER 12, 2020

Inability to maintain context – This is the worst of them all because every time a data set or workload is re-used, you must recreate its context including security, metadata, and governance. Alternatively, you can also spin up a different compute cluster and access the data by using CDP’s Shared Data Experience.

Cloud 104

Cloud Data Warehouse Banking Data 104

Databand.ai

AUGUST 30, 2023

A shorter time-to-value indicates that your organization is efficient at processing and analyzing data for decision-making purposes. Monitoring this metric helps identify bottlenecks in the data pipeline and ensures timely insights are available for business users.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Cloudera

MAY 12, 2021

This category is open to organizations that have tackled transformative business use cases by connecting multiple parts of the data lifecycle to enrich, report, serve, and predict. . DATA FOR ENTERPRISE AI. SECURITY AND GOVERNANCE LEADERSHIP.

Food 70

Food Healthcare Government Medical 70

DataKitchen

AUGUST 16, 2024

At DataKitchen, we think of this is a ‘meta-orchestration’ of the code and tools acting upon the data. Data Pipeline Observability: Optimizes pipelines by monitoring data quality, detecting issues, tracing data lineage, and identifying anomalies using live and historical metadata.

Data Pipeline 50

Data Pipeline Metadata Data Governance Coding 50

DataKitchen

JANUARY 5, 2021

The data teams share a common objective; to create analytics for the (internal or external) customer. Execution of this mission requires the contribution of several groups: data center/IT, data engineering, data science, data visualization, and data governance.

Data Governance 59

Data Governance Government Data Science Metadata 59

Towards Data Science

MAY 19, 2023

Tech Target , defines a data silo as a repository of data controlled by one department or business unit and, therefore, not wholly or easily accessible by other departments within the same organisation. Absence or poor adoption of company-wide guidelines surrounding the creation and deployment of data products.

Pipeline-centric 104

Pipeline-centric Data Software Engineer Software Engineering 104

Monte Carlo

MARCH 13, 2024

Let’s dig in and explore the landscape of the top so-called “data quality tools” — what they are, what they’re not, and whether they’re the right first step towards more reliable data. Governance helps companies set important standards and achieve higher levels of data security, data accessibility, and data quality.

Data Governance 52

Data Governance Government Metadata Data 52

Databand.ai

AUGUST 30, 2023

Poor data quality: The lack of automation and data governance in legacy architectures can lead to data quality issues, such as incomplete, inaccurate, or duplicate data. This requires implementing robust data integration tools and practices, such as data validation, data cleansing, and metadata management.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52