Data Engineering Weekly

OCTOBER 6, 2024

link] Sponsored: Monte Carlo Demo - October You want to deliver reliable business insights to your organization, but that probably means you’re spending a whole lotta time writing a whole lotta of rules. It updates the popular Pirate metrics (AARRR) framework by adding concepts like Attention and Adoption.

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

JUNE 12, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Cloudera

FEBRUARY 8, 2024

Cloudera Data Warehouse (CDW) running Hive has previously supported creating materialized views against Hive ACID source tables. release and the matching CDW Private Cloud Data Services release, Hive also supports creating, using, and rebuilding materialized views for Iceberg table format.

Metadata 89

Metadata Data Warehouse BI AWS 89

Monte Carlo

AUGUST 30, 2024

It’s our goal at Monte Carlo to provide data observability and quality across the enterprise by monitoring every system vital in the delivery of data from source to consumption. We started with popular modern data warehouses and quickly expanded our support as data lakes became data lakehouses.

SQL 40

SQL Engineering Data Lake High Quality Data 40

Data Engineering Podcast

MAY 18, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

NOVEMBER 20, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 130

Data Lake Data Ingestion MongoDB Google Cloud 130

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

RandomTrees

APRIL 18, 2024

By leveraging Python’s capabilities, we can create scripts that analyze Informatica workflows, extract relevant metadata, and generate equivalent Talend jobs. Analysing Informatica Workflows : The first step in the conversion process is to analyze Informatica workflows to extract essential metadata and transformation logic.

Coding 52

Coding Retail Metadata Python 52

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Java 69

Java Metadata PostgreSQL Data Warehouse 69

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. What is your opinion on the relative merits of a data warehouse vs a data lake and are they mutually exclusive?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

MARCH 17, 2023

When implementing a data lakehouse, the table format is a critical piece because it acts as an abstraction layer, making it easy to access all the structured, unstructured data in the lakehouse by any engine or tool, concurrently. Also, the data lake layer is not limited to cloud object stores.

Education 64

Education Unstructured Data Data Lake Data Warehouse 64

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. See our post: Data Lakes vs. Data Warehouses.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Monte Carlo

AUGUST 22, 2023

For example, let’s say a resource monitor has alerted the team that the Snowflake data warehouse, “TRANSFORMING_METRX,” has exceeded its threshold. You can view all the query runtimes for that warehouse during the time period when the alert was triggered. This could drive unwanted costs. Now it’s time to figure out why.

Data Pipeline 52

Data Pipeline Metadata Data Warehouse Utilities 52

Databand.ai

JUNE 28, 2023

This might include processes like data extraction from different sources, data cleansing, data transformation (like aggregation), and loading the data into a database or a data warehouse. Data storage and delivery : Observability continues into the storage and delivery phase. Was this article helpful?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Snowflake

APRIL 17, 2023

What started as a cloud data warehouse evolved into the Data Cloud, which includes the content—data, services, and applications—that customers and partners bring to the cloud. Snowflake makes it easy to build and deliver data and applications to customers across clouds. Oh, and a net promoter score of 72.

Retail 72

Retail Cloud Metadata Data Warehouse 72

dbt Developer Hub

JANUARY 24, 2023

Ensures developers can work quickly by shortening the CI feedback loop Reduce costs in your data warehouse by running only what's been modified I think we can all agree that increasing developer productivity while simultaneously reducing costs is an outcome that every company strives for. A query method on the metadata property.

Metadata 52

Metadata Python Cloud Data Warehouse 52

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Expiring snapshots is a relatively cheap operation and uses metadata to determine newly unreachable files.

Bytes 59

Bytes Metadata Data Lake SQL 59

Data Engineering Weekly

JULY 2, 2023

link] Apache Doris: Building a Data Warehouse for Traditional Industry The blog narates the incremental update and the overall update strategy of a typical legacy datawarehouses. Atlan AI leverages metadata that Atlan captures across the data stack to make AI part of your data stack.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Snowflake

FEBRUARY 29, 2024

This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Caption: A demo Streamlit app visualizes H3 hexagons at different resolutions for the same data. There are 16 layers, or levels, of resolution in H3.

Insurance 90

Insurance SQL BI Consulting 90

Netflix Tech

OCTOBER 27, 2021

The GUI would be targeted at professional data scientists. Instead of a fancy GUI for demos and presentations, we wanted a serious productivity tool with carefully thought-out user workflows that would fit seamlessly into our toolchain of data science. The GUI should integrate well with other GUIs.

Python 91

Python Machine Learning Data Science Project 91

AltexSoft

NOVEMBER 7, 2022

Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. The Good and the Bad of Power BI Data Visualization.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Datakin

OCTOBER 14, 2021

dbt is an amazing way to transform data within a data warehouse. Run models & capture lineage metadata When working with Datakin (or any other OpenLineage backend) it’s important to generate the dbt docs first. Our schema has changed, and we want Datakin to have the latest metadata about tables and columns. %

Google Cloud 52

Google Cloud Datasets Bytes Metadata 52

AltexSoft

OCTOBER 8, 2021

Data integration defines the process of collecting data from a number of disparate source systems and presenting it in a unified form within a centralized location like a data warehouse. So, why is data integration such a big deal? Oracle Data Integrator, IBM InfoSphere, Snaplogic, Xplenty, and.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

Data Engineering Podcast

JANUARY 28, 2018

Contact Info Dat datproject.org Email @dat_project on Twitter Dat Chat Danielle Email @daniellecrobins Joe Email @joeahand on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? So this is really useful for the big data sets, you know, I think the whole thing.

Data 100

Data Project Data Management Electronics 100

Ascend.io

APRIL 7, 2023

Source: Data Mesh Principles and Logical Architecture by Zhamak Dehghani What is a Data Fabric? Data fabric is a centralized platform architecture originating from a curated metadata layer that sits on top of an organization’s data infrastructure. Increasing speed.

Metadata 52

Metadata Datasets Data Governance Government 52

Data Engineering Weekly

JULY 9, 2023

The platform approach to enable the citizen machine learning engineers is a great perspective while building both the Data & ML platform. Architectural patterns like Lambda Architecture and Kappa Architecture emerged to bridge the gap between real-time and batch data processing. Write SQL queries without learning SQL?

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Monte Carlo

APRIL 20, 2023

I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen. And over time, we’ve invested effort into cleaning up our lineages, simplifying our logic,” said SeatGeek Director of Data Engineering, Brian London. Data lineage can help!

Data Warehouse 52

Data Warehouse BI Data Government 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64