phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? They also support ACID transactions, ensuring data integrity and stored data reliability.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Cloudera

OCTOBER 26, 2020

Just like when it comes to data access in business. Enabling data access for end-users so they can drive insight and business value is a typical area of compromise between IT and users. Data access can either be very secure but restrictive or very open yet risky. Quickly onboard data.

Data Security 119

Data Security IT Unstructured Data Government 119

Striim

APRIL 18, 2025

Why AI and Analytics Require Real-Time, High-Quality Data To extract meaningful value from AI and analytics, organizations need data that is continuously updated, accurate, and accessible. Heres why: AI Models Require Clean Data: Machine learning models are only as good as their training data.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Data Engineering Weekly

JUNE 24, 2024

A few highlights from the report Unstructured data goes mainstream. link] Sponsored: 2024 State of Apache Airflow Report Gain access to the latest trends and insights shaping the world of Apache Airflow—the go-to platform for data pipeline development and orchestration.

Data Engineering 110

Data Engineering Data Engineer Engineering Google Cloud 110

Data Engineering Weekly

JULY 21, 2024

The blog is an excellent summary of what one needs to know about Gen-AI to start. link] Manuel Faysse: ColPali - Efficient Document Retrieval with Vision Language Models 👀 80% of enterprise data exists in difficult-to-use formats like HTML, PDF, CSV, PNG, PPTX, and more.

Data Engineering 118

Data Engineering Data Engineer Engineering Generalist 118

phData: Data Engineering

NOVEMBER 1, 2024

In an effort to better understand where data governance is heading, we spoke with top executives from IT, healthcare, and finance to hear their thoughts on the biggest trends, key challenges, and what insights they would recommend. This blog is a collection of those insights, but for the full trendbook, we recommend downloading the PDF.

Government 52

Government Data Governance Finance Metadata 52

Data Engineering Weekly

MARCH 5, 2025

This blog post expands on that insightful conversation, offering a critical look at Iceberg's potential and the hurdles organizations face when adopting it. Initially, catalogs focused on managing metadata for structured data in Iceberg tables.

Hadoop 58

Hadoop Metadata Data Ingestion Data Governance 58

Cloudera

OCTOBER 29, 2020

More importantly, from a security and governance perspective, native integration with CDP means SSO for authentication and seamless integration with Cloudera Shared Data Experience (SDX) to manage user access and governance. With DV, users login with their CDP credentials and start analyzing data that they have access to.

Machine Learning 111

Machine Learning Data Warehouse Unstructured Data Government 111

Cloudera

APRIL 1, 2024

We scored the highest in hybrid, intercloud, and multi-cloud capabilities because we are the only vendor in the market with a true hybrid data platform that can run on any cloud including private cloud to deliver a seamless, unified experience for all data, wherever it lies.

Cloud 105

Cloud Unstructured Data Metadata Government 105

Snowflake

NOVEMBER 2, 2023

Rather than defining schema upfront, a user can decide which data and schema they need for their use case. Snowflake has long supported semi-structured data types and file formats like JSON, XML, Parquet, and more recently storage and processing of unstructured data such as PDF documents, images, videos, and audio files.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Cloudera

JUNE 9, 2022

In the first blog of the Universal Data Distribution blog series , we discussed the emerging need within enterprise organizations to take control of their data flows. controlling distribution while also allowing the freedom and flexibility to deliver the data to different services is more critical than ever. .

Data Collection 94

Data Collection Data Lake Unstructured Data Retail 94

Cloudera

NOVEMBER 10, 2020

Two of the more painful things in your everyday life as an analyst or SQL worker are not getting easy access to data when you need it, or not having easy to use, useful tools available to you that don’t get in your way! HUE’s table browser, with built-in data sampling. Efficient Query Design. Optimization as you go.

SQL 112

SQL Unstructured Data Hadoop Data Lake 112

Cloudera

JANUARY 28, 2022

Then there are the more extensive discussions – scrutiny of the overarching, data strategy questions related to privacy, security, data governance /access and regulatory oversight. These are not straightforward decisions, especially when data breaches always hit the top of the news headlines.

Cloud 115

Cloud Banking Data Governance Government 115

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. PII data) of each data product, and the access rights for each different group of data consumers.

Architecture 124

Architecture Metadata Kafka Government 124

Cloudera

JUNE 7, 2022

In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB.

IT 112

IT Unstructured Data Data Architecture Government 112

Cloudera

DECEMBER 11, 2020

As mentioned in my previous blog on the topic , the recent shift to remote working has seen an increase in conversations around how data is managed. Toolsets and strategies have had to shift to ensure controlled access to data. This is what really stood out about the finalists of the Data Security and Governance category.

Data Lake 98

Data Lake Government Data Security Unstructured Data 98

Cloudera

JULY 29, 2021

Securely protecting healthcare data is critical for your organization’s success, whether data is ingested, streamed and stored in a data platform that runs in the public, private or hybrid cloud. Public, private, hybrid or on-premise data management platform. Be The Change. Security and governance in a hybrid environment.

Healthcare 102

Healthcare Unstructured Data Government Machine Learning 102

Cloudera

MAY 14, 2024

This form of hybrid also goes a level deeper than one may find in a standard hybrid cloud, accounting for the entirety of the data lifecycle, whether that’s the point of ingestion, warehousing, or machine learning—even when that end-to-end data lifecycle is split between entirely different environments. Data comes in many forms.

Cloud 99

Cloud Data Governance Unstructured Data Data Architecture 99

Snowflake

MAY 9, 2024

It started when one capable model suited for text gained mainstream attention, and now, less than 18 months later, there is a long list of commercial and open-source gen AI models are now available, alongside new multimodal models that also understand images and other unstructured data. Ready to dive deeper into gen AI?

Unstructured Data 121

Unstructured Data Manufacturing Retail Data Warehouse 121

Cloudera

JANUARY 5, 2021

The retailer leveraged Cloudera to build an analytics solution for fulfillment delivery that allowed for advanced analytic modeling, A/B testing, and optimization by improved data access of omnichannel orders, logistics, and delivery capacity. Additional retail content can be found at our retail resource kit .

Retail 91

Retail Unstructured Data Big Data Machine Learning 91

Cloudera

AUGUST 13, 2021

Every enterprise is trying to collect and analyze data to get better insights into their business. Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

Cloudera

SEPTEMBER 9, 2022

Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested. An architectural innovation: Cloudera Data Platform (CDP) and Apache Iceberg.

Architecture 102

Architecture Metadata Machine Learning Unstructured Data 102

Cloudera

OCTOBER 28, 2020

To start, they look to traditional financial services data, combining and correlating account activity, borrowing history, core banking, investments, and call center data. While Rabobank has always had access to this data, drawing meaningful insight from it was a different matter. .

Unstructured Data 74

Unstructured Data Banking Machine Learning Media 74

Cloudera

OCTOBER 26, 2020

To learn more, check out the blog post here. . Attribute-based access control and SparkSQL fine-grained access control. Lineage and chain of custody, advanced data discovery and business glossary. Store and access schemas across clusters and rebalance clusters with Cruise Control. Ranger 2.0.

Certification 87

Certification Cloud Kafka Unstructured Data 87

Snowflake

JUNE 28, 2023

To make it even easier and secure for customers to take advantage of leading LLMs, Snowpark Container Services can be used as part of a Snowflake Native App , so customers will be able to get direct access to leading LLMs via the Snowflake Marketplace and installed to run entirely in their Snowflake accounts. Read this blog.

Pipeline-centric 97

Pipeline-centric Unstructured Data Data Government 97

Cloudera

MARCH 17, 2023

Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data.

Education 64

Education Unstructured Data Data Lake Data Warehouse 64

Sync Computing

JANUARY 27, 2025

Data transfers between regions or zones incur additional costs that can outweigh the cost savings, not to mention the impact on performance. Provisioning EC2 instances in the same region as your data is not only important from a cost perspective, it also reduces access latency and increases transfer speed.

AWS 52

AWS Data Lake Machine Learning Big Data 52

Cloudera

SEPTEMBER 15, 2022

Structured data (such as name, date, ID, and so on) will be stored in regular SQL databases like Hive or Impala databases. There are also newer AI/ML applications that need data storage, optimized for unstructured data using developer friendly paradigms like Python Boto API. Ranger policies.

Systems 87

Systems Hadoop Metadata Telecommunication 87

Cloudera

FEBRUARY 7, 2023

Financial inclusion, defined as the availability and accessibility of financial services to underserved communities, is a critical issue facing the banking industry today. Access to financial services and credit can help lift individuals and entire underserved communities out of poverty. According to the World Bank, 1.7

Banking 76

Banking Technology Insurance Education 76

Snowflake

JUNE 28, 2023

By bringing compute closer to the data businesses can eliminate data silos, address security and governance challenges, and optimize operations, leading to enhanced efficiency, all while avoiding the management overhead associated with additional systems and infrastructure.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Cloudera

SEPTEMBER 12, 2022

Organizations don’t know what they have anymore and so can’t fully capitalize on it — the majority of data generated goes unused in decision making. And second, for the data that is used, 80% is semi- or unstructured. Both obstacles can be overcome using modern data architectures, specifically data fabric and data lakehouse.

Unstructured Data 75

Unstructured Data Data Lake Data Architecture Data 75

DoorDash Engineering

APRIL 26, 2023

The company is exploring the use of Generative AI, a subset of Artificial Intelligence that generates novel content based on existing data, and how it can be implemented effectively with consideration for the privacy and security of personal information. In fact, we used generative AI to help edit this blog post!

Food 98

Food Unstructured Data Deep Learning SQL 98

Towards Data Science

OCTOBER 7, 2023

Given LLMs’ capacity to understand and extract insights from unstructured data, businesses are finding value in summarizing, analyzing, searching, and surfacing insights from large amounts of internal information. Let’s explore how a few key sectors are putting gen AI to use.

Unstructured Data 98

Unstructured Data Finance SQL Database 98

Cloudera

JULY 26, 2021

In fact, data product development introduces an additional requirement that wasn’t as relevant in the past as it is today: That of scalability in permissioning and authorization given the number and multitude of different roles of data constituents, both internal and external accessing a data product.

Generalist 119

Generalist Telecommunication Healthcare Data Science 119

Snowflake

JUNE 13, 2024

Like any first step, data ingestion is a critical foundational block. Given the many different ways to ingest data, in this blog we will walk through the various methods, calling out the latest announcements and improvements we’ve made. Ingestion with Snowflake should feel like a breeze.

Data Ingestion 114

Data Ingestion MySQL PostgreSQL Data Pipeline 114

Data Engineering Weekly

SEPTEMBER 18, 2024

Decoupling of Storage and Compute : Data lakes allow observability tools to run alongside core data pipelines without competing for resources by separating storage from compute resources. This opens up new possibilities for monitoring and diagnosing data issues across various sources.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

Cloudera

SEPTEMBER 9, 2020

This blog post will present a simple “hello world” kind of example on how to get data that is stored in S3 indexed and served by an Apache Solr service hosted in a Data Discovery and Exploration cluster in CDP. We will only cover AWS and S3 environments in this blog. You have CLI access to that cluster.

AWS 84

AWS Data Unstructured Data Hadoop 84