Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Data Architect 52

Data Engineering Podcast

AUGUST 13, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day.

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Lyft Engineering

OCTOBER 3, 2023

We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. In future posts we will discuss how Verity addresses data quality elsewhere in our data platform. What and Where is Data Quality?

Big Data 85

Big Data Metadata Data Warehouse Data 85

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Interview Introduction How did you get involved in the area of data management?

Systems 100

Systems Metadata Building MongoDB 100

Netflix Tech

DECEMBER 3, 2022

The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. It lets you create YAML formatted mock data files based on selected tables, columns and a few rows of data from the Netflix data warehouse.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Airbnb Tech

NOVEMBER 28, 2023

However, for all of our uncertified data, which remained the majority of our offline data, we lacked visibility into its quality and didn’t have clear mechanisms for up-leveling it. How could we scale the hard-fought wins and best practices of Midas across our entire data warehouse?

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. You can observe your pipelines with built in metadata search and column level lineage. What is your working definition of that term?

BI 100

BI Data Warehouse Metadata SQL 100

Data Engineering Weekly

JULY 7, 2024

[link] Jack Vanlightly: Understanding Apache Paimon's Consistency Model Part 1 Table formats are definitely a hot topic in the industry now. The author highlights Paimon’s consistency model by examining the metadata model. Apache Paimon is a relatively low-key format spun out of Flink’s table store.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Cloudera

FEBRUARY 8, 2024

Cloudera Data Warehouse (CDW) running Hive has previously supported creating materialized views against Hive ACID source tables. release and the matching CDW Private Cloud Data Services release, Hive also supports creating, using, and rebuilding materialized views for Iceberg table format.

Metadata 89

Metadata Data Warehouse BI AWS 89

Monte Carlo

AUGUST 30, 2023

Your test passes when there are no rows returned, which indicates your data meets your defined conditions. Often, teams run custom data tests as part of a deployment pipeline, or scheduled on production systems via job schedulers like Apache Airflow, dbt Cloud, or via in-built schedulers in your data warehouse solution.

SQL 52

SQL Datasets Database High Quality Data 52

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. What is your working definition of "data governance" and how does that influence your product focus and priorities?

Data Governance 130

Data Governance Government Cloud Building 130

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

MAY 18, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Interview Introduction How did you get involved in the area of data management?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Cloudera

SEPTEMBER 29, 2022

This team or domain expert will be responsible for the data produced by the team. The data itself is then treated as a product. The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. What is a data mesh contract?

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Knowledge Hut

MARCH 7, 2024

This implies two things, one, the data coming from one source is out of date when compared to another source. Two, it creates a commonality of data definitions, concepts, metadata and the like.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Christophe Blefari

FEBRUARY 3, 2023

What's often stays is that AE is between DE and DA, so the definition is often done complementarily to other positions. Microsoft Azure announced managed Airflow — Starting this week you'll be able to launch Apache Airflow within Azure Data Factory. In the end actually, the answer depends. Nothing new under the sun.

BI 130

BI Google Cloud Machine Learning SQL 130

Cloudera

NOVEMBER 5, 2020

Many organizations struggle to meet growing and variable data warehouse demands. This is exactly what Cloudera Data Platform (CDP) provides to the Cloudera Data Warehouse. CDP is a data platform that is optimized for both business units and central IT. . Cloudera Data Warehouse Security.

Cloud 74

Cloud Data Lake Data Warehouse Metadata 74

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed.

Metadata 52

Metadata Data Lake Data Engineering Data Engineer 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project 52

Project Metadata Data Pipeline Data Engineer 52

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. What is a Data Vault model? Pie Insurance , a leading small business insurtech, leverages a data vault 2.0

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Data Engineering Podcast

SEPTEMBER 19, 2021

He also discusses the technical underpinnings of what he is building and how using the data warehouse as a shared resource drastically shortens the time required to see value. That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data.

Machine Learning 100

Machine Learning Data Warehouse Banking Metadata 100

Data Engineering Podcast

NOVEMBER 20, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Monte Carlo

AUGUST 13, 2024

And how should you leverage it as you evaluate different approaches to improving data quality and reliability? The Gartner definition of data observability While Monte Carlo is unquestionably the data observability category’s creator, we’re certainly not the only game in town – and that’s a good thing. Let’s get into it.

Data 52

Data Data Warehouse Data Pipeline Data Architecture 52

Cloudera

JANUARY 21, 2021

While cloud-native, point-solution data warehouse services may serve your immediate business needs, there are dangers to the corporation as a whole when you do your own IT this way. Cloudera Data Warehouse (CDW) is here to save the day! CDW is an integrated data warehouse service within Cloudera Data Platform (CDP).

IT 92

IT Data Lake Data Warehouse Cloud Storage 92

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

ProjectPro

FEBRUARY 8, 2023

It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., Glue uses ETL jobs for extracting data from various AWS cloud services and integrating it into data warehouses and lakes.

AWS 98

AWS Scala Metadata Data Lake 98

Cloudera

MARCH 17, 2023

When implementing a data lakehouse, the table format is a critical piece because it acts as an abstraction layer, making it easy to access all the structured, unstructured data in the lakehouse by any engine or tool, concurrently. Also, the data lake layer is not limited to cloud object stores.

Education 64

Education Unstructured Data Data Lake Data Warehouse 64

Netflix Tech

JULY 22, 2024

Since Netflix’s data tables are housed in a single data warehouse, we believe a single orchestrator should handle all workflows accessing it. A workflow definition is defined in a JSON format. Maestro combines user-supplied fields with those managed by Maestro to form a flexible and powerful orchestration definition.

Machine Learning 102

Machine Learning Coding Data Pipeline Java 102

dbt Developer Hub

JULY 26, 2022

With the dbt Semantic Layer, this definition would live in dbt and the logic to create the dataset for that metric would be consistent across all different consuming experiences. Best of all, definition changes would get reflected in downstream tools, so you no longer need to manually search and update every downstream dependency.

Metadata 52

Metadata BI Datasets SQL 52