Confluent

JUNE 11, 2025

Apache Kafka® Learn more about how Confluent differs from Apache Kafka For Practitioners Discover the platform that is built and designed for those who build For Executives Unlock the value of data across your business Our Customers Explore testimonials and case studies from Confluents customers Products Data Streaming Platform Stream, connect, govern, (..)

Kafka 40

Kafka Telecommunication Manufacturing Retail 40

DataKitchen

DECEMBER 17, 2024

This lack of awareness leads to undetected issues, reactive data cleansing, and costly downstream impacts ( see the webinar: [link] L) The presentation highlights common challenges organizations face when dealing with data quality. The result is a broken, reactive process that fails to prevent data quality issues at their source.

Technology 52

Technology Data Cleanse High Quality Data Metadata 52

Precisely

NOVEMBER 14, 2023

This journey must include a strong data governance framework to align people, processes, and technology, and enable them to understand and trust their data and metadata to achieve their business objectives. How do we align this critical spatial data to our business goals, objectives, metrics, and processes?

Data Governance 120

Data Governance Government Metadata Retail 120

Snowflake

JANUARY 23, 2024

Behind the scenes, Snowpark ML parallelizes data processing operations by taking advantage of Snowflake’s scalable computing platform. For Snowpark ML Operations, the Snowpark Model Registry allows customers to securely manage and execute models in Snowflake, regardless of origin.

Machine Learning 136

Machine Learning Metadata Python Telecommunication 136

DataKitchen

MAY 4, 2023

Webinar Summary: DataOps and Data Mesh Chris Bergh, CEO of DataKitchen, delivered a webinar on two themes – Data Products and Data Mesh. They describe five interfaces to a domain: the width (data), the where (location), the what (description), the how (process), and the who (team). Watch the webinar today!

Raw Data 52

Raw Data Data Datasets Data Engineer 52

Cloudera

MAY 19, 2021

This year, we expanded our partnership with NVIDIA , enabling your data teams to dramatically speed up compute processes for data engineering and data science workloads with no code changes using RAPIDS AI. The script will go through loading RAPIDs libraries then leveraging them to load and processing a datafile. RAPIDS (wall time).

Machine Learning 137

Machine Learning Data Science Datasets Data Lake 137

Snowflake

APRIL 23, 2024

Ensono, a managed service provider and technology adviser, joined the initial preview phase of the Snowflake Connector for ServiceNow and began using it as part of its customer portal and data warehouse modernization project (watch their Show Me Your Architecture webinar here ).

Data Warehouse 110

Data Warehouse Consulting Data Integration Metadata 110

Snowflake

FEBRUARY 8, 2024

Snowflake’s Global Head of Manufacturing Tim Long says smart manufacturing — the use of advanced technologies to improve the efficiency of traditional processes — is a “huge area of interest” that industry leaders can supercharge with data and AI. Data collaboration is the process of gathering and sharing data from various sources.

Manufacturing 101

Manufacturing Metadata Data Technology 101

Cloudera

JUNE 25, 2019

Cloudera Unveils Industry’s First Enterprise Data Cloud in Webinar. Over 2000 customers and partners joined us in this live webinar featuring a first-look at our upcoming cloud-native CDP services. Cloudera received extraordinary interest in CDP from participants, exemplified by the more than 300 questions posed throughout the webinar.

Cloud 87

Cloud Entertainment Government Machine Learning 87

Snowflake

AUGUST 22, 2023

The data product lifecycle includes the following stages: Discovery Design Development Deployment Let’s take a look at what they entail and the roles that lead and support each of them: Discovery starts the process. A prioritization matrix can help formalize this process. High-cost products that deliver high value are OK.

Metadata 91

Metadata Data AWS Business Analyst 91

Cloudera

DECEMBER 15, 2022

Compute engines in these CDP data services can access and process data sets in the Iceberg tables concurrently, with shared security and governance provided by our unique Cloudera Shared Data Experience ( SDX ). Only metadata will be regenerated. Metadata management . Amazingly fast table migration. ORC open file format support.

Cloud 80

Cloud Metadata Data Warehouse Google Cloud 80

Snowflake

JULY 11, 2024

As a result, operational processes are evolving to accommodate increased content production and distribution channels, including streaming platforms, social media networks, podcast platforms and virtual reality environments. A streamlined media supply chain helps fuel commercial success.

Media 88

Media Entertainment Cloud Metadata 88

Cloudera

AUGUST 30, 2022

Solutions that support MDAs are purpose-built for data collection, processing, and sharing. Integration, metadata and governance capabilities glue the individual components together.”. A data mesh supports distributed, domain-specific data consumers and views data as a product, with each domain handling its own data pipelines.

Data Architecture 79

Data Architecture Architecture Data Lake NoSQL 79

Cloudera

AUGUST 16, 2021

While NiFi nodes can be added to an existing cluster, it is a multi-step process that requires organizations to set up constant monitoring of resource usage, detect when there is enough demand to scale, automate the provisioning of a new node with the required software and set up the security configuration. and later).

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

Cloudera

FEBRUARY 8, 2024

We populated the tables using INSERT-SELECT statements by reading from text format source tables but they can be populated through any ETL process. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Such a query pattern is quite common in BI queries.

Metadata 91

Metadata Data Warehouse BI AWS 91

Cloudera

JUNE 18, 2022

It added metadata that described the logical and physical layout of the data, enabling cost-based optimizers, dynamic partition pruning, and a number of key performance improvements targeted at SQL analytics. The first generation of the Hive Metastore attempted to address the performance considerations to run SQL efficiently on a data lake.

Data Lake 94

Data Lake Data Warehouse BI SQL 94

Cloudera

DECEMBER 9, 2022

This allows developers to make changes to their processing logic on the fly while running some test data through their flow and validating that their changes work as intended. Once you have retrieved the data, NiFi stores it in a queue, which allows you to explore the content and metadata attributes of the events.

Designing 100

Designing Coding Google Cloud AWS 100

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Java 70

Java Metadata PostgreSQL Data Warehouse 70

Data Engineering Weekly

AUGUST 18, 2024

The recent Stack Overflow survey echoes the statement that the usage of AI tools is gaining popularity in the development process. The need to adopt software development practices in the ETL process is much higher, as the success of AI-driven applications depends on data quality. link] Sponsored: Airflow 2.10 has landed!

Data Engineering 64

Data Engineering Data Engineer Engineering Kafka 64

Cloudera

APRIL 3, 2023

The table metadata is stored next to the data files under a metadata directory, which allows multiple engines to use the same table simultaneously. CDW separates the compute (Virtual Warehouses) and metadata (DB catalogs) by running them in independent Kubernetes pods. Read why the future of data lakehouses is open.

Data Warehouse 82

Data Warehouse Java Metadata Data 82

Data Engineering Weekly

FEBRUARY 18, 2024

Learn about Cube, the universal semantic layer, in an upcoming technical webinar. Register for our webinar to explore Cube Cloud and learn about the convenient UI for easier data modeling. I believe the data ownership problem is much deeper than simple metadata management.

Data Engineer 64

Data Engineer Data Engineering Engineering Data 64

Snowflake

FEBRUARY 29, 2024

This process of placing shapes over a real or imaginary map in a game has parallels to the real world — we call it creating a spatial grid. This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Line 10: the output table is specified.

Insurance 107

Insurance SQL Consulting BI 107

DataKitchen

OCTOBER 11, 2021

The biggest challenge is broken data pipelines due to highly manual processes. Figure 1: Example data pipeline with manual processes. There are numerous challenges with this process, as described below. There’s a fear of making changes to the process as it might break production. When can you declare it done?

Engineering 61

Engineering Business Analyst BI Metadata 61

Cloudera

JULY 14, 2023

It could be much easier to simply stop all those jobs rather than allowing them to continue during the migration process. They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Silectis

FEBRUARY 25, 2021

In this blog post (and the accompanying webinar recording) I provide a high-level overview lineage and how it applies in practice. Lineage is history – What is the change log for any element of metadata? LINEAGE IN CONTEXT Data lineage doesn’t exist in a vacuum, it is one of many tools one can use during the data engineering process.

ETL Tools 52

ETL Tools Metadata Data Data Engineer 52

Cloudera

JUNE 4, 2018

The solution to this massive data challenge embedded the Aspire Content Processing Framework into the Cloudera Enterprise Data Hub as a Cloudera Parcel – a binary distribution format containing the program files, along with additional metadata used by Cloudera Manager. compliance reporting.

Pharmaceutical 46

Pharmaceutical Unstructured Data Electronics Metadata 46

Rockset

SEPTEMBER 26, 2023

The Rockset console where you can setup the Confluent Cloud integration Real-time updates and metadata filtering in Rockset While Confluent delivers the real-time data for AI applications, the other half of the AI equation is a serving layer capable of handling stringent latency and scale requirements.

Kafka 64

Kafka Metadata Cloud Database 64

Databand.ai

NOVEMBER 4, 2022

In a recent webinar with IBM, we dug into why data observability is so important, what’s needed for data observability, and how Databand can help. Identifying a run duration issue with Databand and DataStage Next, we have a process that ran in DataStage for which we had previously set an alert if the run exceeds 120 seconds.

Data 52

Data Data Pipeline Retail Metadata 52

Ascend.io

JULY 23, 2024

This article (based on the webinar below) explores how data engineering teams can leverage AI and automation to enhance productivity and tackle current challenges. Initially, data teams focused their effort on scaling storage and processing capabilities. Learn how we use metadata to automate 90% of manual data pipeline maintenance.]

Data Engineer 52

Data Engineer Data Engineering Engineering Metadata 52

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Weekly

DECEMBER 4, 2022

The author discusses the need for richer metadata to support complex data lineage and evolving privacy requirements. Upsolver SQLake lets you process fast-moving data by simply writing a SQL query. link] Barr Moses: What’s Next for Data Engineering in 2023? Pipelines for data in motion can quickly turn into DAG hell.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Monte Carlo

JANUARY 30, 2024

This case study is based on information shared in recent Snowflake webinars and Summit presentations. At JetBlue, we use dimension tracking to monitor the health of the data attributes we would expect to see in our business process. It monitors the distribution of values and is really useful. You can proactively receive notifications.

Data 64

Data SQL Data Engineer Data Engineering 64

Monte Carlo

MARCH 9, 2023

That is what JetBlue did as described by data scientist Derrick Olson in a recent Snowflake webinar. For a real Monte Carlo example, one of our production models makes use of a “seconds since last metadata refresh” feature. But we know that this value should never be negative, otherwise we’d be somehow measuring data from the future.

Engineering 52

Engineering Machine Learning Data Pipeline Data Science 52

Data Engineering Weekly

OCTOBER 23, 2022

The Data Engineering Weekly even published a special Metadata Edition focusing on the historical development of the Data Catalog. link] It is almost two years since we published the metadata edition, but I keep thinking back. I'm one of the early advocates for Data Catalogs and am excited about the possibility of Data Catalogs.

Data Engineer 52

Data Engineer Data Engineering Engineering Deep Learning 52

Knowledge Hut

SEPTEMBER 29, 2023

The Azure Data Engineer Certification test evaluates one's capacity for organizing and putting into practice data processing, security, and storage, as well as their capacity for keeping track of and maximizing data processing and storage. Why Should You Get an Azure Data Engineer Certification?

Certification 52

Certification Data Engineer Data Engineering Engineering 52

Knowledge Hut

NOVEMBER 28, 2023

The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Kafka is great for ETL and provides memory buffers that provide process reliability and resilience.

Data Engineer 52

Data Engineer Data Engineering Engineering Generalist 52

Databand.ai

DECEMBER 29, 2022

She’s trained thousands on how to strategically use the power of data visualization to enhance the decision-making process. Beyond her hands-on work, Colleen is determined to make engineering organizations better for both humans and business through mentoring, leadership, and streamlining processes.

Consulting 52

Consulting BI Data Science Data Governance 52