Snowflake

AUGUST 22, 2023

The data product lifecycle includes the following stages: Discovery Design Development Deployment Let’s take a look at what they entail and the roles that lead and support each of them: Discovery starts the process. A prioritization matrix can help formalize this process. High-cost products that deliver high value are OK.

Metadata 72

Metadata Data AWS Business Analyst 72

Data Engineering Weekly

AUGUST 18, 2024

The recent Stack Overflow survey echoes the statement that the usage of AI tools is gaining popularity in the development process. The need to adopt software development practices in the ETL process is much higher, as the success of AI-driven applications depends on data quality. link] Sponsored: Airflow 2.10 has landed!

Data Engineer 62

Data Engineer Data Engineering Engineering Kafka 62

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Java 70

Java Metadata PostgreSQL Data Warehouse 70

Ascend.io

JULY 23, 2024

This article (based on the webinar below) explores how data engineering teams can leverage AI and automation to enhance productivity and tackle current challenges. Initially, data teams focused their effort on scaling storage and processing capabilities. Learn how we use metadata to automate 90% of manual data pipeline maintenance.]

Data Engineer 52

Data Engineer Data Engineering Engineering Metadata 52

Cloudera

FEBRUARY 8, 2024

We populated the tables using INSERT-SELECT statements by reading from text format source tables but they can be populated through any ETL process. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Such a query pattern is quite common in BI queries.

Metadata 90

Metadata Data Warehouse BI AWS 90

Cloudera

AUGUST 30, 2022

Solutions that support MDAs are purpose-built for data collection, processing, and sharing. Integration, metadata and governance capabilities glue the individual components together.”. A data mesh supports distributed, domain-specific data consumers and views data as a product, with each domain handling its own data pipelines.

Data Architecture 79

Data Architecture Architecture Data Lake NoSQL 79

Data Engineering Weekly

FEBRUARY 18, 2024

Learn about Cube, the universal semantic layer, in an upcoming technical webinar. Register for our webinar to explore Cube Cloud and learn about the convenient UI for easier data modeling. I believe the data ownership problem is much deeper than simple metadata management.

Data Engineer 62

Data Engineer Data Engineering Engineering Data 62

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineer 52

Data Engineer Data Engineering Engineering Datasets 52

Cloudera

MAY 19, 2021

This year, we expanded our partnership with NVIDIA , enabling your data teams to dramatically speed up compute processes for data engineering and data science workloads with no code changes using RAPIDS AI. The script will go through loading RAPIDs libraries then leveraging them to load and processing a datafile. RAPIDS (wall time).

Machine Learning 137

Machine Learning Datasets Data Science Raw Data 137

Cloudera

JULY 14, 2023

It could be much easier to simply stop all those jobs rather than allowing them to continue during the migration process. They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Cloudera

JUNE 25, 2019

Cloudera Unveils Industry’s First Enterprise Data Cloud in Webinar. Over 2000 customers and partners joined us in this live webinar featuring a first-look at our upcoming cloud-native CDP services. Cloudera received extraordinary interest in CDP from participants, exemplified by the more than 300 questions posed throughout the webinar.

Cloud 87

Cloud Entertainment Government Machine Learning 87

Cloudera

JUNE 18, 2022

It added metadata that described the logical and physical layout of the data, enabling cost-based optimizers, dynamic partition pruning, and a number of key performance improvements targeted at SQL analytics. The first generation of the Hive Metastore attempted to address the performance considerations to run SQL efficiently on a data lake.

Data Lake 93

Data Lake Data Warehouse BI SQL 93

Databand.ai

NOVEMBER 4, 2022

In a recent webinar with IBM, we dug into why data observability is so important, what’s needed for data observability, and how Databand can help. Identifying a run duration issue with Databand and DataStage Next, we have a process that ran in DataStage for which we had previously set an alert if the run exceeds 120 seconds.

Data 52

Data Data Pipeline Retail Metadata 52

Cloudera

DECEMBER 9, 2022

This allows developers to make changes to their processing logic on the fly while running some test data through their flow and validating that their changes work as intended. Once you have retrieved the data, NiFi stores it in a queue, which allows you to explore the content and metadata attributes of the events.

Designing 100

Designing Coding Google Cloud AWS 100

Cloudera

APRIL 3, 2023

The table metadata is stored next to the data files under a metadata directory, which allows multiple engines to use the same table simultaneously. CDW separates the compute (Virtual Warehouses) and metadata (DB catalogs) by running them in independent Kubernetes pods. Read why the future of data lakehouses is open.

Data Warehouse 81

Data Warehouse Java Metadata Data 81

Rockset

SEPTEMBER 26, 2023

The Rockset console where you can setup the Confluent Cloud integration Real-time updates and metadata filtering in Rockset While Confluent delivers the real-time data for AI applications, the other half of the AI equation is a serving layer capable of handling stringent latency and scale requirements.

Metadata 64

Metadata Kafka Cloud Database 64

Silectis

FEBRUARY 25, 2021

In this blog post (and the accompanying webinar recording) I provide a high-level overview lineage and how it applies in practice. Lineage is history – What is the change log for any element of metadata? LINEAGE IN CONTEXT Data lineage doesn’t exist in a vacuum, it is one of many tools one can use during the data engineering process.

ETL Tools 52

ETL Tools Metadata Data Data Engineer 52

Data Engineering Weekly

DECEMBER 4, 2022

The author discusses the need for richer metadata to support complex data lineage and evolving privacy requirements. Upsolver SQLake lets you process fast-moving data by simply writing a SQL query. link] Barr Moses: What’s Next for Data Engineering in 2023? Pipelines for data in motion can quickly turn into DAG hell.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Monte Carlo

JANUARY 30, 2024

This case study is based on information shared in recent Snowflake webinars and Summit presentations. At JetBlue, we use dimension tracking to monitor the health of the data attributes we would expect to see in our business process. It monitors the distribution of values and is really useful. You can proactively receive notifications.

Data 64

Data SQL Data Engineer Data Engineering 64

DataKitchen

OCTOBER 11, 2021

The biggest challenge is broken data pipelines due to highly manual processes. Figure 1: Example data pipeline with manual processes. There are numerous challenges with this process, as described below. There’s a fear of making changes to the process as it might break production. When can you declare it done?

Engineering 61

Engineering Business Analyst BI Metadata 61

Monte Carlo

MARCH 9, 2023

That is what JetBlue did as described by data scientist Derrick Olson in a recent Snowflake webinar. For a real Monte Carlo example, one of our production models makes use of a “seconds since last metadata refresh” feature. But we know that this value should never be negative, otherwise we’d be somehow measuring data from the future.

Engineering 52

Engineering Machine Learning Data Pipeline Data Science 52

Data Engineering Weekly

OCTOBER 23, 2022

The Data Engineering Weekly even published a special Metadata Edition focusing on the historical development of the Data Catalog. link] It is almost two years since we published the metadata edition, but I keep thinking back. I'm one of the early advocates for Data Catalogs and am excited about the possibility of Data Catalogs.

Data Engineer 52

Data Engineer Data Engineering Engineering Deep Learning 52

Cloudera

AUGUST 16, 2021

While NiFi nodes can be added to an existing cluster, it is a multi-step process that requires organizations to set up constant monitoring of resource usage, detect when there is enough demand to scale, automate the provisioning of a new node with the required software and set up the security configuration. and later).

Cloud 121

Cloud Unstructured Data Utilities Metadata 121

AltexSoft

NOVEMBER 7, 2022

The tool represents processes in the form of directed acyclic graphs which visualize casual relationships between tasks and the order of their execution. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. Metadata database. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Snowflake

FEBRUARY 29, 2024

This process of placing shapes over a real or imaginary map in a game has parallels to the real world — we call it creating a spatial grid. This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Line 10: the output table is specified.

Insurance 88

Insurance SQL BI Consulting 88

Cloudera

JUNE 4, 2018

The solution to this massive data challenge embedded the Aspire Content Processing Framework into the Cloudera Enterprise Data Hub as a Cloudera Parcel – a binary distribution format containing the program files, along with additional metadata used by Cloudera Manager. compliance reporting.

Pharmaceutical 45

Pharmaceutical Unstructured Data Electronics Metadata 45

Knowledge Hut

NOVEMBER 28, 2023

The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Kafka is great for ETL and provides memory buffers that provide process reliability and resilience.

Data Engineer 52

Data Engineer Data Engineering Engineering Generalist 52

Monte Carlo

JUNE 14, 2023

Data pipeline architecture is the process of designing how data is surfaced from its source system to the consumption layer. It’s important to understand most data pipelines aren’t a linear movement of data from source A to target B, but rather consist of a series of highly complex and interdependent processes.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Monte Carlo

JUNE 20, 2023

In this post, we will explore the process of implementation and go further by exploring what else is possible. In our example, the team has decided to track data freshness checks based on the processing timestamp attribute present in the dataset that is served by the data product: processing_timestamp.

Building 52

Building Data Datasets High Quality Data 52

Databand.ai

DECEMBER 29, 2022

She’s trained thousands on how to strategically use the power of data visualization to enhance the decision-making process. Beyond her hands-on work, Colleen is determined to make engineering organizations better for both humans and business through mentoring, leadership, and streamlining processes.

BI 52

BI Consulting Data Science Data Governance 52

ProjectPro

JULY 23, 2021

Real-time Data Processing Application 7. The migration process from an existing platform to an AWS-based solution is also easy—security, reliability, and ease of use are other attributes attached to AWS projects and applications. Cloud Computing technologies are now an integral part of business processes and operations.

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Knowledge Hut

SEPTEMBER 29, 2023

The Azure Data Engineer Certification test evaluates one's capacity for organizing and putting into practice data processing, security, and storage, as well as their capacity for keeping track of and maximizing data processing and storage. Why Should You Get an Azure Data Engineer Certification?

Certification 52

Certification Data Engineer Data Engineering Engineering 52

Monte Carlo

MAY 17, 2023

Know When To Fix Vs. Refactor Data Pipelines Improve DataOps Processes 37. Quick time to detection and resolution is important because it makes the process of backfilling the correct data less painful.” That is why the majority of organizations have extensive code review and other quality assurance processes in place.

Data 52

Data Data Pipeline Data Engineer Data Engineering 52

Monte Carlo

MAY 17, 2023

Know When To Fix Vs. Refactor Data Pipelines Improve DataOps Processes 37. Quick time to detection and resolution is important because it makes the process of backfilling the correct data less painful.” That is why the majority of organizations have extensive code review and other quality assurance processes in place.

Data Pipeline 52

Data Pipeline Data Engineer Data Engineering Data 52