Cloudera

FEBRUARY 8, 2024

We populated the tables using INSERT-SELECT statements by reading from text format source tables but they can be populated through any ETL process. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Such a query pattern is quite common in BI queries.

Metadata 96

Metadata Data Warehouse BI AWS 96

Snowflake

FEBRUARY 8, 2024

Snowflake’s Global Head of Manufacturing Tim Long says smart manufacturing — the use of advanced technologies to improve the efficiency of traditional processes — is a “huge area of interest” that industry leaders can supercharge with data and AI. Data collaboration is the process of gathering and sharing data from various sources.

Manufacturing 87

Manufacturing Metadata Data Technology 87

Cloudera

AUGUST 30, 2022

Solutions that support MDAs are purpose-built for data collection, processing, and sharing. Integration, metadata and governance capabilities glue the individual components together.”. A data mesh supports distributed, domain-specific data consumers and views data as a product, with each domain handling its own data pipelines.

Data Architecture 84

Data Architecture Architecture Data Lake NoSQL 84

Cloudera

JUNE 18, 2022

It added metadata that described the logical and physical layout of the data, enabling cost-based optimizers, dynamic partition pruning, and a number of key performance improvements targeted at SQL analytics. The first generation of the Hive Metastore attempted to address the performance considerations to run SQL efficiently on a data lake.

Data Lake 98

Data Lake Data Warehouse BI SQL 98

Cloudera

AUGUST 16, 2021

While NiFi nodes can be added to an existing cluster, it is a multi-step process that requires organizations to set up constant monitoring of resource usage, detect when there is enough demand to scale, automate the provisioning of a new node with the required software and set up the security configuration. and later).

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

Snowflake

JULY 11, 2024

As a result, operational processes are evolving to accommodate increased content production and distribution channels, including streaming platforms, social media networks, podcast platforms and virtual reality environments. A streamlined media supply chain helps fuel commercial success.

Media 76

Media Entertainment Cloud Metadata 76

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Java 75

Java Metadata PostgreSQL Data Warehouse 75

Snowflake

AUGUST 22, 2023

The data product lifecycle includes the following stages: Discovery Design Development Deployment Let’s take a look at what they entail and the roles that lead and support each of them: Discovery starts the process. A prioritization matrix can help formalize this process. High-cost products that deliver high value are OK.

Metadata 75

Metadata Data AWS Business Analyst 75

Cloudera

DECEMBER 9, 2022

This allows developers to make changes to their processing logic on the fly while running some test data through their flow and validating that their changes work as intended. Once you have retrieved the data, NiFi stores it in a queue, which allows you to explore the content and metadata attributes of the events.

Designing 105

Designing Coding Google Cloud AWS 105

Cloudera

APRIL 3, 2023

The table metadata is stored next to the data files under a metadata directory, which allows multiple engines to use the same table simultaneously. CDW separates the compute (Virtual Warehouses) and metadata (DB catalogs) by running them in independent Kubernetes pods. Read why the future of data lakehouses is open.

Data Warehouse 87

Data Warehouse Java Metadata Data 87

Data Engineering Weekly

FEBRUARY 18, 2024

Learn about Cube, the universal semantic layer, in an upcoming technical webinar. Register for our webinar to explore Cube Cloud and learn about the convenient UI for easier data modeling. I believe the data ownership problem is much deeper than simple metadata management.

Data Engineering 64

Data Engineering Data Engineer Engineering Data 64

Data Engineering Weekly

AUGUST 18, 2024

The recent Stack Overflow survey echoes the statement that the usage of AI tools is gaining popularity in the development process. The need to adopt software development practices in the ETL process is much higher, as the success of AI-driven applications depends on data quality. link] Sponsored: Airflow 2.10 has landed!

Data Engineering 64

Data Engineering Data Engineer Engineering Kafka 64

Cloudera

JULY 14, 2023

It could be much easier to simply stop all those jobs rather than allowing them to continue during the migration process. They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 75

Metadata Data Warehouse Big Data Ecosystem Java 75

DataKitchen

OCTOBER 11, 2021

The biggest challenge is broken data pipelines due to highly manual processes. Figure 1: Example data pipeline with manual processes. There are numerous challenges with this process, as described below. There’s a fear of making changes to the process as it might break production. When can you declare it done?

Engineering 64

Engineering BI Business Analyst Metadata 64

DataKitchen

DECEMBER 17, 2024

This lack of awareness leads to undetected issues, reactive data cleansing, and costly downstream impacts ( see the webinar: [link] L) The presentation highlights common challenges organizations face when dealing with data quality. The result is a broken, reactive process that fails to prevent data quality issues at their source.

Technology 52

Technology Data Cleanse High Quality Data Metadata 52

Ascend.io

JULY 23, 2024

This article (based on the webinar below) explores how data engineering teams can leverage AI and automation to enhance productivity and tackle current challenges. Initially, data teams focused their effort on scaling storage and processing capabilities. Learn how we use metadata to automate 90% of manual data pipeline maintenance.]

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Silectis

FEBRUARY 25, 2021

In this blog post (and the accompanying webinar recording) I provide a high-level overview lineage and how it applies in practice. Lineage is history – What is the change log for any element of metadata? LINEAGE IN CONTEXT Data lineage doesn’t exist in a vacuum, it is one of many tools one can use during the data engineering process.

ETL Tools 52

ETL Tools Metadata Data Data Engineering 52

Snowflake

FEBRUARY 29, 2024

This process of placing shapes over a real or imaginary map in a game has parallels to the real world — we call it creating a spatial grid. This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Line 10: the output table is specified.

Insurance 91

Insurance SQL BI Consulting 91

Cloudera

JUNE 4, 2018

The solution to this massive data challenge embedded the Aspire Content Processing Framework into the Cloudera Enterprise Data Hub as a Cloudera Parcel – a binary distribution format containing the program files, along with additional metadata used by Cloudera Manager. compliance reporting.

Pharmaceutical 48

Pharmaceutical Unstructured Data Electronics Metadata 48

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Monte Carlo

JANUARY 30, 2024

This case study is based on information shared in recent Snowflake webinars and Summit presentations. At JetBlue, we use dimension tracking to monitor the health of the data attributes we would expect to see in our business process. It monitors the distribution of values and is really useful. You can proactively receive notifications.

Data 64

Data SQL Data Engineering Data Engineer 64

Rockset

SEPTEMBER 26, 2023

The Rockset console where you can setup the Confluent Cloud integration Real-time updates and metadata filtering in Rockset While Confluent delivers the real-time data for AI applications, the other half of the AI equation is a serving layer capable of handling stringent latency and scale requirements.

Metadata 64

Metadata Kafka Cloud Database 64

Databand.ai

NOVEMBER 4, 2022

In a recent webinar with IBM, we dug into why data observability is so important, what’s needed for data observability, and how Databand can help. Identifying a run duration issue with Databand and DataStage Next, we have a process that ran in DataStage for which we had previously set an alert if the run exceeds 120 seconds.

Data 52

Data Data Pipeline Retail Metadata 52

Data Engineering Weekly

DECEMBER 4, 2022

The author discusses the need for richer metadata to support complex data lineage and evolving privacy requirements. Upsolver SQLake lets you process fast-moving data by simply writing a SQL query. link] Barr Moses: What’s Next for Data Engineering in 2023? Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

MARCH 9, 2023

That is what JetBlue did as described by data scientist Derrick Olson in a recent Snowflake webinar. For a real Monte Carlo example, one of our production models makes use of a “seconds since last metadata refresh” feature. But we know that this value should never be negative, otherwise we’d be somehow measuring data from the future.

Engineering 52

Engineering Machine Learning Data Pipeline Data Science 52

Data Engineering Weekly

OCTOBER 23, 2022

The Data Engineering Weekly even published a special Metadata Edition focusing on the historical development of the Data Catalog. link] It is almost two years since we published the metadata edition, but I keep thinking back. I'm one of the early advocates for Data Catalogs and am excited about the possibility of Data Catalogs.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

AltexSoft

NOVEMBER 7, 2022

The tool represents processes in the form of directed acyclic graphs which visualize casual relationships between tasks and the order of their execution. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. Metadata database. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Knowledge Hut

NOVEMBER 28, 2023

The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Kafka is great for ETL and provides memory buffers that provide process reliability and resilience.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

Databand.ai

DECEMBER 29, 2022

She’s trained thousands on how to strategically use the power of data visualization to enhance the decision-making process. Beyond her hands-on work, Colleen is determined to make engineering organizations better for both humans and business through mentoring, leadership, and streamlining processes.

BI 52

BI Consulting Data Science Data Governance 52

Monte Carlo

JUNE 14, 2023

Data pipeline architecture is the process of designing how data is surfaced from its source system to the consumption layer. It’s important to understand most data pipelines aren’t a linear movement of data from source A to target B, but rather consist of a series of highly complex and interdependent processes.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Monte Carlo

JUNE 20, 2023

In this post, we will explore the process of implementation and go further by exploring what else is possible. In our example, the team has decided to track data freshness checks based on the processing timestamp attribute present in the dataset that is served by the data product: processing_timestamp.

Building 52

Building Data Datasets High Quality Data 52

ProjectPro

JULY 23, 2021

Real-time Data Processing Application 7. The migration process from an existing platform to an AWS-based solution is also easy—security, reliability, and ease of use are other attributes attached to AWS projects and applications. Cloud Computing technologies are now an integral part of business processes and operations.

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Knowledge Hut

SEPTEMBER 29, 2023

The Azure Data Engineer Certification test evaluates one's capacity for organizing and putting into practice data processing, security, and storage, as well as their capacity for keeping track of and maximizing data processing and storage. Why Should You Get an Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Monte Carlo

MAY 17, 2023

Know When To Fix Vs. Refactor Data Pipelines Improve DataOps Processes 37. Quick time to detection and resolution is important because it makes the process of backfilling the correct data less painful.” That is why the majority of organizations have extensive code review and other quality assurance processes in place.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Know When To Fix Vs. Refactor Data Pipelines Improve DataOps Processes 37. Quick time to detection and resolution is important because it makes the process of backfilling the correct data less painful.” That is why the majority of organizations have extensive code review and other quality assurance processes in place.

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Data 52