Data Engineering Podcast

NOVEMBER 27, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Missing data? images, documents, etc.) Atlan is the metadata hub for your data ecosystem. Stale dashboards?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. My advice on this point is to learn from others.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

phData: Data Engineering

SEPTEMBER 19, 2023

Want to learn more about data governance? Check out our Data Governance on Snowflake blog! Metadata Management Data modeling methodologies help in managing metadata within the data lake. Metadata describes the characteristics, attributes, and context of the data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Netflix Tech

DECEMBER 3, 2022

The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. It lets you create YAML formatted mock data files based on selected tables, columns and a few rows of data from the Netflix data warehouse.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Christophe Blefari

JUNE 16, 2023

Data warehouses are mutable, this is one of the many root causes proposed by Lucas. The past often changes, whether because of code or data. Data Documentation 101: Why? — Marie wrote best practices for establishing complete and reliable data documentation. This is metrics drift.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Snowflake

APRIL 23, 2024

Ensono, a managed service provider and technology adviser, joined the initial preview phase of the Snowflake Connector for ServiceNow and began using it as part of its customer portal and data warehouse modernization project (watch their Show Me Your Architecture webinar here ).

Data Warehouse 99

Data Warehouse Data Integration Consulting Metadata 99

Airbnb Tech

NOVEMBER 28, 2023

However, achieving the full data quality criteria required by Midas demands significant cross-functional investment to design, develop, validate, and maintain the necessary data assets and documentation. While this made sense for our most critical data, pursuing such rigorous standards at scale presented challenges.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

dbt Developer Hub

NOVEMBER 13, 2023

This mostly just worked – as long as you didn't have anything more complex than a CI job mixed in with a couple of production jobs – because important constructs like deferral in CI and documentation were only ever tied to a single job. To achieve this, configure your data warehouse, git repo and dbt Cloud account as shown above.

Cloud 52

Cloud Database-centric Metadata Data Warehouse 52

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Cloudera

AUGUST 30, 2022

It outlines a scenario in which “recently married people might want to change their names on their driver’s licenses or other documentation. That should be easy, but when agencies don’t share data or applications, they don’t have a unified view of people. Forrester ).

Data Architecture 78

Data Architecture Architecture Data Lake NoSQL 78

Data Engineering Weekly

AUGUST 25, 2024

The author summarizes the learning from the Google Search Engine paper leak and various public hearing documents from antitrust cases. Here’s what I discovered Apache Parquet becomes the defacto columnar storage for LakeHouse formats, and it is a must for data engineers to know its internals.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

ProjectPro

FEBRUARY 8, 2023

It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., Glue uses ETL jobs for extracting data from various AWS cloud services and integrating it into data warehouses and lakes.

AWS 98

AWS Scala Metadata Data Lake 98

Edureka

AUGUST 14, 2024

It is the open-source software that increased the Paraquet data assignments. This is a scalable metadata system because it handles transaction performance and fills the files with it. Updating and modifying Delta Lake tables The Delta Lake atomic transactions enhance many options for upgrading and updating metadata.

Data Lake 40

Data Lake Metadata SQL Data Pipeline 40

Monte Carlo

FEBRUARY 12, 2024

To differentiate and expand the usefulness of these models, organizations must augment them with first-party data – typically via a process called RAG (retrieval augmented generation). Today, this first-party data mostly lives in two types of data repositories.

Database 72

Database Unstructured Data Data Pipeline Metadata 72

Cloudera

NOVEMBER 5, 2020

Many organizations struggle to meet growing and variable data warehouse demands. This is exactly what Cloudera Data Platform (CDP) provides to the Cloudera Data Warehouse. CDP is a data platform that is optimized for both business units and central IT. . Cloudera Data Warehouse Security.

Cloud 74

Cloud Data Lake Data Warehouse Metadata 74

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Monte Carlo

MAY 18, 2023

A question to ask is how will these data lineage source solutions account for dbt models, Fivetran connectors, and their relationship to tables? If our organization decided to pack up and migrate from one table format or data warehouse to another, would all the work we put into maintaining our open source data lineage solution for naught?

Metadata 64

Metadata Data Lake Government Data Governance 64

phData: Data Engineering

OCTOBER 12, 2022

Hosted Doc Site for Documentation One of the most powerful features of dbt can be the documentation you generate. This documentation can give different users insight into where data came from, what the profile of the data is, what the SQL looked like, and the DAG to know where the data is being used.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Towards Data Science

FEBRUARY 6, 2024

Order snapshots are stored in my own development area (image by the author) To prevent my extractions from impacting performance on the operational side, I queried this data regularly and stored it in a persistent staging area (PSA) within my data warehouse. Metadata update Data products need to be understandable.

Systems 78

Systems Raw Data Metadata Data Cleanse 78

Towards Data Science

DECEMBER 1, 2023

This tool automates ELT (Extract, Load, Transform) process, integrating your data from the source system of Google Calendar to our Snowflake data warehouse. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 69

Data Engineering Data Engineer Project Engineering 69

dbt Developer Hub

DECEMBER 19, 2023

The documentation will help you familiarize yourself with how our users interact with dbt. Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. New to dbt Cloud? ​

Metadata 52

Metadata Cloud Accessible Accessibility 52

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Cloudera

SEPTEMBER 29, 2022

Data mesh conceptual hierarchy. Instead of having a central team that manages all the data for a company, the thinking is that the responsibility of generating, curating, documenting, updating, and managing data should be distributed across the company based on whichever team is best suited to produce and own that data.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. dbt used in transformation pipelines on data warehouses (Image source: [link].

Data Warehouse 79

Data Warehouse Data Lake Government High Quality Data 79

Ascend.io

FEBRUARY 28, 2024

This process reduces noise in the data, which is crucial for the effectiveness of AI algorithms, especially in complex predictive models and deep learning applications. Such comprehensive metadata management is crucial in adhering to privacy and compliance standards, safeguarding AI operations against potential legal and ethical pitfalls.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. Snowflake is a DataLake Platform Snowflake is moving beyond a SQL data warehouse.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

dbt Developer Hub

OCTOBER 30, 2023

However, one of the questions I’ve received most frequently is guidance around “when” to clone that goes beyond the documentation on “how” to clone. that leverages native zero-copy clone functionality on supported warehouses to copy entire schemas for free, almost instantly. How is it different from deferral? How is this possible?

BI 52

BI Datasets Metadata Data Warehouse 52

Monte Carlo

FEBRUARY 26, 2024

A data catalog is a constantly updated inventory of the universe of data assets within an organization. It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems.

Metadata 52

Metadata Government Data Data Governance 52

Grouparoo

FEBRUARY 9, 2022

The testing process is often performed during the initial setup of a data warehouse after new data sources are added to a pipeline and after data integration and migration projects. ETL testing can be challenging since most ETL systems process large volumes of heterogeneous data. Extract data from source systems.

Process 52

Process ETL System Data Warehouse Metadata 52

Monte Carlo

JUNE 28, 2022

Traditionally, data lakes held raw data in its native format and were known for their flexibility, speed, and open source ecosystem. By design, data was less structured with limited metadata and no ACID properties. Since then, Databricks has aggressively moved toward allowing users to add more structure to their data.

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

Monte Carlo

APRIL 4, 2023

Data anomalies and incidents can be introduced across any point of this pipeline, which can make root cause analysis a fragmented and frustrating multi-tab process. Now teams can instantly get full visibility into how these systems may be impacting their data assets, all in a single pane of glass. A modified dbt model?

BI 64

BI Data Ingestion Data Pipeline Metadata 64

Cloudera

JANUARY 21, 2021

While cloud-native, point-solution data warehouse services may serve your immediate business needs, there are dangers to the corporation as a whole when you do your own IT this way. Cloudera Data Warehouse (CDW) is here to save the day! CDW is an integrated data warehouse service within Cloudera Data Platform (CDP).

IT 92

IT Data Lake Data Warehouse Cloud Storage 92

Monte Carlo

JULY 25, 2023

Data curation is the process of transforming and enriching larger amounts of raw data into smaller, more widely accessible subsets of data that provide additional value to the organization or the intended use case. For smaller organizations, this may be the only level of data curation. Here are some example architectures.

Raw Data 52

Raw Data Data Warehouse Data Architecture 52