Snowflake

MAY 15, 2024

By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly. The post Snowflake Invests in Metaplane for Deep, End-to-End Observability in the Data Cloud appeared first on Snowflake.

Cloud 126

Cloud Metadata Data Pipeline Government 126

Cloudera

AUGUST 6, 2024

Many companies adopted the public cloud, but very few organizations will ever move everything to the cloud, or to a single cloud. The future for most data teams will be multi-cloud and hybrid. And for that future to be a reality, data teams must shift their attention to metadata, the new turf war for data.

Metadata 82

Metadata Government Datasets Architecture 82

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Review the Upgrade document topic for the supported upgrade paths.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML. Arroyo, a stream-processing platform, rebuilt their engine using DataFusion.

Metadata 272

Metadata Data Data Warehouse Software Engineer 272

Data Engineering Podcast

MARCH 27, 2022

The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project. Can you describe what Privacera is and the story behind it?

Data Governance 130

Data Governance Government Cloud Building 130

dbt Developer Hub

APRIL 20, 2025

These tools can be called by LLM systems to learn about your data and metadata. No - there is functionality for both dbt Cloud and dbt Core users included in the MCP. Over time, Cloud-specific services will be built into the MCP server where they provide differentiated value.

Structured Data 98

Structured Data SQL BI Project 98

Cloudera

MAY 2, 2022

There are many reasons to deploy a hybrid cloud architecture — not least cost, performance, reliability, security, and control of infrastructure. But increasingly at Cloudera, our clients are looking for a hybrid cloud architecture in order to manage compliance requirements.

Cloud 104

Cloud Government Healthcare Metadata 104

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Cloudera

AUGUST 26, 2021

The object store is readily available alongside HDFS in CDP (Cloudera Data Platform) Private Cloud Base 7.1.3+. Before we jump into the data ingestion step, here is a quick overview of how Ozone manages its metadata namespace through volumes, buckets and keys. . Ozone Namespace Overview. Data ingestion through ‘s3’.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Data Engineering Podcast

MAY 22, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Data orchestration engines are a natural point for generating and taking advantage of rich metadata. Last year you founded Union to offer a managed version of Flyte.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. The Snowflake Data Cloud offering enables access to all of your customer data alongside infinite compute power to model it.

Cloud 116

Cloud Building Insurance Data Engineering 116

Cloudera

JANUARY 26, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 73

Metadata Data Governance Government Management 73

Snowflake

APRIL 16, 2025

Process all your data where it already lives Fragmented data environments and complex cloud architectures impede efficiency and innovation. Other examples include retailers who integrate product photo metadata with transaction histories to gain deeper insights of how visuals influence purchase decisions.

Data Analysis 75

Data Analysis Unstructured Data Manufacturing Retail 75

Cloudera

AUGUST 16, 2021

We just announced Cloudera DataFlow for the Public Cloud (CDF-PC), the first cloud-native runtime for Apache NiFi data flows. The need for a cloud-native Apache NiFi service. A technical look at Cloudera DataFlow for the Public Cloud. This is what Cloudera DataFlow for the Public Cloud offers to NiFi users.

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

Precisely

NOVEMBER 18, 2024

Cloud modernization presents challenges. Yoğurtçu also says that pervasive cloud modernization is a growing issue. With the rise of cloud-based data management, many organizations face the challenge of accessing both on-premises and cloud-based data. Focus on metadata management.

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Snowflake

NOVEMBER 2, 2023

From the start, the Snowflake platform has been delivered as a service, consisting of optimized storage, elastic multi-cluster compute, and cloud services. The post Announcing New Innovations for Data Warehouse, Data Lake, and Data Lakehouse in the Data Cloud appeared first on Snowflake.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Netflix Tech

SEPTEMBER 24, 2021

As an example, cloud-based post-production editing and collaboration pipelines demand a complex set of functionalities, including the generation and hosting of high quality proxy content. The inspection stage examines the input media for compliance with Netflix’s delivery specifications and generates rich metadata.

Cloud 96

Cloud Bytes Cloud Storage Media 96

Christophe Blefari

JUNE 21, 2024

Snowflake and Databricks have the same goal, both are selling a cloud on top of classic 1  cloud vendors. Both companies have added Data and AI to their slogan, Snowflake used to be The Data Cloud and now they're The AI Data Cloud. But there are a few issues with Parquet.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Cloudera

APRIL 1, 2024

We’re excited to share that Gartner has recognized Cloudera as a Visionary among all vendors evaluated in the 2023 Gartner® Magic Quadrant for Cloud Database Management Systems. Download the complimentary 2023 Gartner Magic Quadrant for Cloud Database Management Systems report.

Cloud 105

Cloud Unstructured Data Metadata Government 105

Cloudera

DECEMBER 6, 2024

The AI Forecast: Data and AI in the Cloud Era , sponsored by Cloudera, aims to take an objective look at the impact of AI on business, industry, and the world at large. It could be metadata that you weren’t capturing before. Artificial Intelligence promises to transform lives and business as we know it.

Manufacturing 82

Manufacturing Retail Finance Metadata 82

Data Engineering Weekly

MARCH 5, 2025

This ecosystem includes: Catalogs: Services that manage metadata about Iceberg tables (e.g., Maintenance Processes: Operations that optimize Iceberg tables, such as compacting small files and managing metadata. Metadata Overhead: Iceberg relies heavily on metadata to track table changes and enable features like time travel.

Hadoop 58

Hadoop Metadata Data Ingestion Data Governance 58

Towards Data Science

AUGUST 15, 2023

Teams quickly understand how hyper elastic compute and storage services can enable them to handle more diverse data types at a previously unheard of volume and velocity, but they don’t always understand the impact of the cloud to their workflows. Image courtesy of Shane Murray and the author. Let’s dive in. Let’s dive in. For this week.

Cloud 98

Cloud Metadata Machine Learning Media 98

Cloudera

JULY 15, 2021

The release of Cloudera Data Platform (CDP) Private Cloud Base edition provides customers with a next generation hybrid cloud architecture. Private Cloud Base Overview. The storage layer for CDP Private Cloud, including object storage. Traditional data clusters for workloads not ready for cloud. Edge or Gateway.

Architecture 102

Architecture Cloud Kafka Hadoop 102

Data Engineering Podcast

AUGUST 13, 2022

The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. In fact, while only 3.5% That’s where our friends at Ascend.io

Metadata 100

Metadata MongoDB MySQL Scala 100

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. . -> Airflow and Luigi -> Dagster, Prefect, Lyft, etc. Go to [dataengineeringpodcast.com/materialize]([link] Support Data Engineering Podcast

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Snowflake

FEBRUARY 2, 2023

During a cloud migration to Snowflake’s Data Cloud, businesses often struggle to know what data they have on premises, what they should migrate, and in what order. And because of this, many organizations fall into a “lift and shift” approach, where everything is simply copied over—as it messily stands—to the cloud.

Cloud 98

Cloud Data Data Governance Government 98

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 122

Building Metadata Cloud Storage AWS 122

phData: Data Engineering

NOVEMBER 1, 2024

Organizations also need a better understanding of how LLMs are trained, especially with external vendors or public cloud environments. Additionally, how your data might be used across vendor LLMs when used in public cloud scenarios will be important to ensure that valuable IP or PII does not leave your platform.

Government 52

Government Data Governance Finance Metadata 52

François Nguyen

MARCH 22, 2021

It will be illustrated with our technical choices and the services we are using in the Google Cloud Platform. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

phData: Data Engineering

NOVEMBER 8, 2024

Then, we add another column called HASHKEY , add more data, and locate the S3 file containing metadata for the iceberg table. Hence, the metadata files record schema and partition changes, enabling systems to process data with the correct schema and partition structure for each relevant historical dataset.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Precisely

FEBRUARY 26, 2025

Automated metadata management – AI-generated catalog asset descriptions significantly reduce manual efforts and improve metadata quality – enabling teams to focus on more strategic tasks. With the ability to turn functionality on or off based on business requirements, you gain full control over when and how AI is applied.

Data Integration 64

Data Integration Data Management Management Data Governance 64

Data Engineering Podcast

FEBRUARY 27, 2022

As organizations start to adopt cloud technologies they need a way to manage the distribution, discovery, and collaboration of data across their operating environments. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud.

Unstructured Data 100

Unstructured Data Cloud Management Metadata 100

Data Engineering Podcast

SEPTEMBER 1, 2021

In recent months the community has focused their efforts on making it the fastest possible option for running your analytics in the cloud. In this episode Dipti Borkar discusses the work that she and her team are doing at Ahana to simplify the work of running your own PrestoDB environment in the cloud.

Data Lake 100

Data Lake Cloud AWS SQL 100

Cloudera

DECEMBER 15, 2022

Today, we are thrilled to share some new advancements in Cloudera’s integration of Apache Iceberg in CDP to help accelerate your multi-cloud open data lakehouse implementation. Multi-cloud deployment with CDP public cloud. Multi-cloud capability is now available for Apache Iceberg in CDP. Advanced capabilitie.

Cloud 80

Cloud Metadata Data Warehouse Google Cloud 80

Cloudera

FEBRUARY 17, 2022

CDP Private Cloud Base is an on-premises version of Cloudera Data Platform (CDP). Provides a consistent experience on Public Cloud, Multi-Cloud, and Private Cloud deployments. One of our previous blogs discussed the four paths to get from legacy platforms to CDP Private Cloud Base. How to upgrade from HDP to CDP.

Cloud 101

Cloud Data Professional Services Database 101

Snowflake

NOVEMBER 2, 2023

Snowflake’s single, cross-cloud governance model has always been a powerful differentiator, enabling customers to manage their increasingly complex data ecosystems with simplicity and ease. As a result, Snowflake is enhancing its governance capabilities that thousands of customers already rely on through Snowflake Horizon.

Metadata 131

Metadata Government AWS Medical 131

Precisely

OCTOBER 16, 2023

Rapid advancements in digital technologies are transforming cloud-based computing and cloud analytics. How can today’s mainframe shops navigate these changes, address the challenges of mainframe integration , and come up with practical strategies for merging their mainframe data with cloud environments?

Cloud 72

Cloud Cloud Computing Metadata Data Governance 72