Data Engineering Podcast

JUNE 17, 2021

Summary Working with unstructured data has typically been a motivation for a data lake. Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable.

Unstructured Data 100

Unstructured Data Data Warehouse Metadata Media 100

Snowflake

NOVEMBER 11, 2024

Snowflake will be introducing new multimodal SQL functions (private preview soon) that enable data teams to run analytical workflows on unstructured data, such as images. With these functions, teams can run tasks such as semantic filters and joins across unstructured data sets using familiar SQL syntax.

Unstructured Data 94

Unstructured Data SQL AWS Healthcare 94

Snowflake

JUNE 12, 2024

A closer look at Snowflake Arctic-TILT Snowflake’s Arctic-TILT, the model powering Document AI, is a Snowflake built LLM that leverages a proprietary and unique transformer architecture, tailored to understand and extract data from documents. Learn more here about Snowflake Cortex AI and Snowflake Copilot.

Unstructured Data 98

Unstructured Data Finance Insurance Manufacturing 98

Snowflake

MARCH 6, 2025

Large language models (LLMs) are transforming how we extract value from this data by running tasks from categorization to summarization and more. While AI has proved that real-time conversations in natural language are possible with LLMs, extracting insights from millions of unstructured data records using these LLMs can be a game changer.

Unstructured Data 88

Unstructured Data Medical Media Data Workflow 88

Snowflake

NOVEMBER 4, 2024

Hear Dr. Andrew Ng talk about AI, agents and how to mobilize unstructured data Prominent AI researcher, founder of DeepLearning.AI Andrew Ng talk about AI, agents and how to mobilize unstructured data Prominent AI researcher, founder of DeepLearning.AI Does it have to change how your team works too?

Unstructured Data 101

Unstructured Data Python Machine Learning Data Pipeline 101

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 102

Architecture Metadata Machine Learning Unstructured Data 102

Data Engineering Weekly

OCTOBER 27, 2024

Astasia Myers: The three components of the unstructured data stack LLMs and vector databases significantly improved the ability to process and understand unstructured data. The blog is an excellent summary of the existing unstructured data landscape. link] Alibaba: Evolution of Flink 2.0

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57

Cloudera

JANUARY 7, 2025

Leveraging Clouderas hybrid architecture, the organization optimized operational efficiency for diverse workloads, providing secure and compliant operations across jurisdictions while improving response times for public health initiatives. This transition streamlined data analytics workflows to accommodate significant growth in data volumes.

Cloud 82

Cloud Government AWS Unstructured Data 82

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts. Let’s examine a few.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Snowflake

MARCH 19, 2025

Attendees will discover how to accelerate their critical business workflows with the right data, technology and ecosystem access. Explore AI and unstructured data processing use cases with proven ROI: This year, retailers and brands will face intense pressure to demonstrate tangible returns on their AI investments.

Retail 83

Retail Media Entertainment Unstructured Data 83

Cloudera

JULY 29, 2022

This does not mean ‘one of each’ – a public cloud data strategy and an on-prem data strategy. Rather, it means a holistic and comprehensive enterprise data strategy, spanning both, supported by a modern data architecture. . The telco industry has also increased its spend by 48% on similar initiatives. .

Data Architecture 89

Data Architecture Architecture Unstructured Data Government 89

Striim

AUGUST 31, 2023

Kappa Architectures are becoming a popular way of unifying real-time (streaming) and historical (batch) analytics giving you a faster path to realizing business value with your pipelines. Kappa Architecture combines streaming and batch while simultaneously turning data warehouses and data lakes into near real-time sources of truth.

Data Integration 69

Data Integration Architecture Amazon Web Services Machine Learning 69

RandomTrees

NOVEMBER 25, 2024

The AI can then propose adjustments to the network architecture so that operators can alleviate congestion on their network. The considerable amount of unstructured data required Random Trees to create AI models that ensure privacy and data handling.

Telecommunication 52

Telecommunication IT Unstructured Data Data Mining 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs.

Data Lake 110

Data Lake Data Warehouse Cloud Unstructured Data 110

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

phData: Data Engineering

NOVEMBER 1, 2024

Strong data governance also lays the foundation for better model performance, cost efficiency, and improved data quality, which directly contributes to regulatory compliance and more secure AI systems. Quotes GenAI and LLM will impact data platforms as they need a bigger amount of data to better train the models.

Government 52

Government Data Governance Finance Metadata 52

Cloudera

NOVEMBER 7, 2023

Are you struggling to manage the ever-increasing volume and variety of data in today’s constantly evolving landscape of modern data architectures? OBS buckets provide rich storage for media files and other unstructured data enabling exploration of unstructured data.

Systems 105

Systems Hadoop Unstructured Data Media 105

Cloudera

JUNE 7, 2022

In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB. Where data flows, ideas follow.

IT 112

IT Unstructured Data Data Architecture Government 112

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

MARCH 3, 2025

System Sprawl – Currently, there is not what we would call an industry standard architecture, although hints are emerging. The data + AI stack is actually four separate stacks coming together: structured data, unstructured data, AI and oftentimes the SaaS stack.

Unstructured Data 52

Unstructured Data Data Cloud Computing Banking 52

Monte Carlo

MAY 28, 2024

A data ingestion architecture is the technical blueprint that ensures that every pulse of your organization’s data ecosystem brings critical information to where it’s needed most. A typical data ingestion flow. Popular Data Ingestion Tools Choosing the right ingestion technology is key to a successful architecture.

Data Ingestion 52

Data Ingestion Architecture Designing Hadoop 52

Data Engineering Weekly

MARCH 5, 2025

The Catalog Conundrum: Beyond Structured Data The role of the catalog is evolving. Initially, catalogs focused on managing metadata for structured data in Iceberg tables. However, the vision is expanding to encompass unstructured data (images, videos, audio) and AI models.

Hadoop 57

Hadoop Metadata Data Ingestion Data Governance 57

U-Next

AUGUST 25, 2022

An open-spurce NoSQL database management program, MongoDB architecture, is used as an alternative to traditional RDMS. MongoDB is built to fulfil the needs of modern apps, with a technical base that allows you through: The document data model demonstrates the most effective approach to work with data. How Does It Function?

MongoDB 40

MongoDB Architecture NoSQL MySQL 40

AltexSoft

SEPTEMBER 23, 2021

One of the innovative ways to address this problem is to build a data hub — a platform that unites all your information sources under a single umbrella. This article explains the main concepts of a data hub, its architecture, and how it differs from data warehouses and data lakes. What is Data Hub?

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Snowflake

JANUARY 18, 2024

Here are five reasons to bring your ERP data to Snowflake: 1. A robust, flexible architecture Snowflake’s unique architecture is designed to handle the full volume, velocity and variety of data without making manufacturers deal with downtime for upgrades or compute changes.

Manufacturing 106

Manufacturing Unstructured Data Cloud Architecture 106

Cloudera

MAY 14, 2024

To attain that level of data quality, a majority of business and IT leaders have opted to take a hybrid approach to data management, moving data between cloud, on-premises -or a combination of the two – to where they can best use it for analytics or feeding AI models. Data comes in many forms. Let’s dive deeper.

Cloud 99

Cloud Data Governance Unstructured Data Data Architecture 99

Snowflake

SEPTEMBER 19, 2023

AI unlocks new data use cases. With the ability to handle unstructured data types and larger volumes of data, AI gives us the tools to tackle more complex, exciting problems. And that was the basis of this new architecture of how neural networks are arranged. Some takeaways?

Machine Learning 114

Machine Learning Unstructured Data Data Analytics Government 114

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases. What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Snowflake

FEBRUARY 13, 2025

Interoperable storage: Snowflake enables customers to access and process structured, semi-structured and unstructured data seamlessly, without silos or delays. Unique automations and optimizations include encryption by default, built-in storage compression and fast access to data even at petabyte scale.

Management 76

Management Government Cloud Unstructured Data 76

AltexSoft

AUGUST 29, 2023

Data lakes emerged as expansive reservoirs where raw data in its most natural state could commingle freely, offering unprecedented flexibility and scalability. This article explains what a data lake is, its architecture, and diverse use cases. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Traditional data warehouse platform architecture. Data lake architecture example.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Cloudera

SEPTEMBER 12, 2022

And second, for the data that is used, 80% is semi- or unstructured. Combining and analyzing both structured and unstructured data is a whole new challenge to come to grips with, let alone doing so across different infrastructures.

Unstructured Data 75

Unstructured Data Data Lake Data Architecture Data 75

Zalando Engineering

JUNE 26, 2017

The main goal of IR systems is to provide a communication layer that enables customers to establish a retrieval dialogue with underlying data.

Architecture 40

Architecture Unstructured Data Machine Learning Data Ingestion 40

Data Engineering Podcast

SEPTEMBER 14, 2020

With Tree Schema you can create your data catalog and have it fully populated in under five minutes when using one of the many automated adapters that can connect directly to your data stores. What are the benefits and tradeoffs of in-memory computation for data-intensive workloads? How is Hazelcast implemented?

Process 100

Process Unstructured Data Metadata Data Engineering 100