Data Engineering Podcast

FEBRUARY 27, 2022

Summary There are a wealth of options for managing structured and textual data, but unstructured binary data assets are not as well supported across the ecosystem. Can you describe what Komprise is and the story behind it? Who are the target customers of the Komprise platform?

Unstructured Data 100

Unstructured Data Cloud Management Metadata 100

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts. Challenges Faced by AI Data Engineers Just because “AI” involved doesn’t mean all the challenges go away!

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

AltexSoft

MAY 12, 2023

In today’s data-driven world, organizations amass vast amounts of information that can unlock significant insights and inform decision-making. A staggering 80 percent of this digital treasure trove is unstructured data, which lacks a pre-defined format or organization. What is unstructured data?

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

Snowflake

FEBRUARY 13, 2025

Lastly, companies have historically collaborated using inefficient and legacy technologies requiring file retrieval from FTP servers, API scraping and complex data pipelines. These processes were costly and time-consuming and also introduced governance and security risks, as once data is moved, customers lose all control.

Management 93

Management Government Cloud Unstructured Data 93

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Snowflake

APRIL 22, 2025

To pile onto the challenge, the vast majority of any companys data is unstructured think PDFs, videos and images. So to capitalize on AI's potential, you need a platform that supports structured and unstructured data without compromising accuracy, quality and governance. 51% say data preparation is too hard.

IT 59

IT Unstructured Data Government Data Warehouse 59

ProjectPro

JUNE 6, 2025

According to the Data Management Body of Knowledge, a Data Architect "provides a standard common business vocabulary, expresses strategic requirements, outlines high-level integrated designs to meet those requirements, and aligns with enterprise strategy and related business architecture."

Data Architect 40

Data Architect Data Mining Programming Language Java 40

ProjectPro

JUNE 6, 2025

If you're wondering how the ETL process can drive your company to a new era of success, this blog will help you discover what use cases of ETL make it a critical component in many data management and analytic systems. The ETL approach can minimize your effort while maximizing the value of the data gathered.

BI 45

BI ETL Tools Retail Healthcare 45

phData: Data Engineering

NOVEMBER 1, 2024

Quotes It's extremely important because many of the Gen AI and LLM applications take an unstructured data approach, meaning many of the tools require you to give the tools full access to your data in an unrestricted way and let it crawl and parse it completely.

Government 52

Government Data Governance Finance Metadata 52

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

ProjectPro

JUNE 6, 2025

To be more specific, ETL developers are responsible for the following tasks: Creating a Data Warehouse - ETL developers create a data warehouse specifically designed to meet the demands of a company after determining the needs. Build effective data pipelines by accessing numerous data sources with Big Data tools and technologies.

Data Engineer 59

Data Engineer Data Engineering Engineering ETL Tools 59

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Use cases change, needs change, technology changes – and therefore data infrastructure should be able to scale and evolve with change.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Snowflake

NOVEMBER 11, 2024

What if you could streamline your efforts while still building an architecture that best fits your business and technology needs? Snowflake is committed to doing just that by continually adding features to help our customers simplify how they architect their data infrastructure. Here’s a closer look.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

ProjectPro

JUNE 6, 2025

Business Intelligence and Artificial Intelligence are popular technologies that help organizations turn raw data into actionable insights. While both BI and AI provide data-driven insights, they differ in how they help businesses gain a competitive edge in the data-driven marketplace.

Business Intelligence 56

Business Intelligence BI Data Mining Raw Data 56

ProjectPro

JUNE 6, 2025

Big data analytics market is expected to be worth $103 billion by 2023. We know that 95% of companies cite managing unstructured data as a business problem. of companies plan to invest in big data and AI. million managers and data analysts with deep knowledge and experience in big data.

Big Data 45

Big Data Big Data Skills Scala Hadoop 45

ProjectPro

JUNE 6, 2025

These businesses need data engineers who can use technologies for handling data quickly and effectively since they have to manage potentially profitable real-time data. These platforms facilitate effective data management and other crucial Data Engineering activities.

Data Engineer 40

Data Engineer Data Engineering Engineering Google Cloud 40

ProjectPro

JUNE 6, 2025

Big Data refers to the massive volumes of data which is no longer possible to manage using traditional software applications. Automated tools are developed as part of the Big Data technology to handle the massive volumes of varied data sets. It will also assist you in building more effective data pipelines.

Big Data 40

Big Data Data Engineer Data Engineering Engineering 40

ProjectPro

JUNE 6, 2025

With the increasing demand for data storage and management, cloud-based solutions, such as Azure Blob Storage, have become essential to modern business operations. Azure Blob Storage provides businesses a scalable and cost-effective way to manage huge amounts of unstructured data, such as images, multimedia files, and documents.

Cloud Storage 40

Cloud Storage Cloud Unstructured Data Data Lake 40

Cloudera

JUNE 7, 2022

In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB. The future is hybrid data, embrace it.

IT 112

IT Unstructured Data Government Data Architecture 112

phData: Data Engineering

NOVEMBER 8, 2024

Track data files within the table along with their column statistics. Open table formats enable efficient data management and retrieval by storing these files chronologically, with a history of DDL and DML actions and an index of data file locations. Log all Inserts, Updates, and Deletes (DML) applied to the table.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Cloudera

FEBRUARY 8, 2021

Adding more wires and throwing more compute hardware to the problem is simply not viable considering the cost and complexities of today’s connected cars or the additional demands designed into electric cars (like battery management systems and eco-trip planning).

Manufacturing 108

Manufacturing Machine Learning Electronics Data Ingestion 108

ProjectPro

JUNE 6, 2025

Microsoft's Azure Data Lake is designed to simplify big data analytics and storage. It streamlines the process of ingesting and storing your data while accelerating the execution of batch, streaming, and interactive analytics. It can effectively store organized, semi-structured, and unstructured data.

Data Lake 40

Data Lake Hadoop Big Data SQL 40

Data Engineering Weekly

OCTOBER 11, 2024

The data world is abuzz with speculation about the future of data engineering and the successor to the celebrated modern data stack. While the modern data stack has undeniably revolutionized data management with its cloud-native approach, its complexities and limitations are becoming increasingly apparent.

Manufacturing 109

Manufacturing Transportation Data Warehouse Unstructured Data 109

Snowflake

NOVEMBER 29, 2023

Data cloud technology can accelerate FAIRification of the world’s biomedical patient data. Next-generation sequencing (NGS) technology has dramatically dropped the price of genomic sequencing, from about $1 million in 2007 to $600 today per whole genome sequencing (WGS).

Metadata 117

Metadata Healthcare Medical Data Storage 117

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing.

Data Management 52

Data Management Management Data Lake Data Governance 52

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing.

Data Management 52

Data Management Management Data Lake Data Governance 52

ProjectPro

JUNE 6, 2025

.” said the McKinsey Global Institute (MGI) in its executive overview of last month's report: "The Age of Analytics: Competing in a Data-Driven World." 2016 was an exciting year for big data with organizations developing real-world solutions with big data analytics making a major impact on their bottom line.

Big Data 40

Big Data Hadoop Data Lake Data Governance 40

ProjectPro

JUNE 6, 2025

Data engineers and architects can provide high-quality data useful for executive decisions. Data Engineer vs Data Architect - Who Does What? Data Architect Roles and Responsibilities Data engineers collect, store, and organize data for analysis by wrangling it and fixing data anomalies.

Data Architect 40

Data Architect Data Engineer Data Engineering Building 40

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing.

Data Management 52

Data Management Management Data Lake Data Governance 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Management Data Lake Data Warehouse 52

Data Engineering Podcast

FEBRUARY 24, 2019

Summary Deep learning is the latest class of technology that is gaining widespread interest. As data engineers we are responsible for building and managing the platforms that power these models. What are some ways that we can use deep learning as part of the data management process?

Deep Learning 100

Deep Learning Data Engineer Data Engineering Engineering 100

Snowflake

APRIL 8, 2024

In fact, 8 of the 10 startups in our semi-finalist list plan to use one or both of these technologies in their offerings. Their analytics-first approach to healthcare leverages AI-powered insights and workflows through natively integrated data management, analytics and care management solutions.

Pipeline-centric 131

Pipeline-centric Food Healthcare Unstructured Data 131

Cloudera

MAY 14, 2024

Hybrid cloud plays a central role in many of today’s emerging innovations—most notably artificial intelligence (AI) and other emerging technologies that create new business value and improve operational efficiencies. But getting there requires data, and a lot of it. Data comes in many forms. What do we mean by ‘true’ hybrid?

Cloud 99

Cloud Data Governance Unstructured Data Government 99

ProjectPro

JUNE 6, 2025

NoSQL databases are the new-age solutions to distributed unstructured data storage and processing. The speed, scalability, and fail-over safety offered by NoSQL databases are needed in the current times in the wake of Big Data Analytics and Data Science technologies.

NoSQL 40

NoSQL Database Hadoop Big Data 40

ProjectPro

JUNE 6, 2025

Here are the five essential components: Data Source Diversity: Zero ETL can handle various data types, including structured and unstructured data from databases, web services, and APIs. This flexibility allows organizations to integrate data from multiple sources without upfront standardization.

Data Analytics 40

Data Analytics MySQL PostgreSQL Data Lake 40