Data Engineering Podcast

AUGUST 14, 2021

In this episode Davit Buniatyan, founder and CEO of Activeloop, explains why he is spending his time and energy on building a platform to simplify the work of getting your unstructured data ready for machine learning. What is the process for sourcing, processing, and storing data to be used by Hub/Activeloop?

Unstructured Data 100

Unstructured Data Machine Learning Data Lake SQL 100

Snowflake

NOVEMBER 11, 2024

At BUILD 2024, we announced several enhancements and innovations designed to help you build and manage your data architecture on your terms. Data stewards can also set up Request for Access (private preview) by setting a new visibility property on objects along with contact details so the right person can easily be reached to grant access.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

AltexSoft

MAY 12, 2023

In today’s data-driven world, organizations amass vast amounts of information that can unlock significant insights and inform decision-making. A staggering 80 percent of this digital treasure trove is unstructured data, which lacks a pre-defined format or organization. What is unstructured data?

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

Precisely

MARCH 9, 2023

Data warehouse vs. data lake, each has their own unique advantages and disadvantages; it’s helpful to understand their similarities and differences. In this article, we’ll focus on a data lake vs. data warehouse. It is often used as a foundation for enterprise data lakes.

Data Lake 59

Data Lake Data Warehouse Hadoop Raw Data 59

phData: Data Engineering

NOVEMBER 8, 2024

Note : Cloud Data warehouses like Snowflake and Big Query already have a default time travel feature. However, this feature becomes an absolute must-have if you are operating your analytics on top of your data lake or lakehouse. It can also be integrated into major data platforms like Snowflake. Contact phData Today!

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Sync Computing

NOVEMBER 7, 2024

While data warehouses are still in use, they are limited in use-cases as they only support structured data. Data lakes add support for semi-structured and unstructured data, and data lakehouses add further flexibility with better governance in a true hybrid solution built from the ground-up.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Hevo

APRIL 20, 2024

In the early 2000s, organizations started dealing with more semi-structured and unstructured data, which consisted of images, videos, log files, text, and sensor data. They needed a storage solution that was more flexible than a data warehouse.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Data 52

Data Engineering Weekly

FEBRUARY 9, 2025

[link] QuantumBlack: Solving data quality for gen AI applications Unstructured data processing is a top priority for enterprises that want to harness the power of GenAI. It brings challenges in data processing and quality, but what data quality means in unstructured data is a top question for every organization.

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Monte Carlo

JULY 15, 2024

We recently spoke with Killian Farrell , Principal Data Scientist at insurance startup AssuranceIQ to learn how his team built an LLM-based product to structure unstructured data and score customer conversations for developing sales and customer support teams. Read on to find out what they did, and what they learned!

Unstructured Data 94

Unstructured Data Insurance Data Lake Structured Data 94

Grouparoo

JANUARY 11, 2022

This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle. There are two main options available, a data lake and a data warehouse. What is a Data Warehouse? What is a Data Lake?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. What is Data Lake? .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Monte Carlo

MARCH 12, 2025

Data lakes turned into swamps , pipelines burst, and just when you thought youd earned a degree in hydrology, someone leaned in and whispered: Delta Lake. Are we building data dams next? Lets break it down and see when a plain data lake works and when youll want the extra reliability of Delta Lake.

Data Lake 52

Data Lake Raw Data Media Unstructured Data 52

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Monte Carlo

AUGUST 25, 2023

That’s why it’s essential for teams to choose the right architecture for the storage layer of their data stack. But, the options for data storage are evolving quickly. Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Snowflake

FEBRUARY 13, 2025

Interoperable storage: Snowflake enables customers to access and process structured, semi-structured and unstructured data seamlessly, without silos or delays. Unique automations and optimizations include encryption by default, built-in storage compression and fast access to data even at petabyte scale.

Management 93

Management Government Cloud Unstructured Data 93

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind. Data discovery tools and platforms can help.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Government 52

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently. And what is the reason for that?

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Architecture Business Intelligence Unstructured Data 52

Monte Carlo

JUNE 2, 2021

As companies leverage more and more data to drive decision-making and maintain their competitive edge, it’s crucial that this data is accurate and reliable. Over the last few years, data lakes have emerged as a must-have for the modern data stack. What is data observability?

Data Lake 52

Data Lake Machine Learning Unstructured Data Datasets 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake? Consistency of data throughout the data lake.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Weekly

SEPTEMBER 18, 2024

The Rise of Data Observability Data observability has become increasingly critical as companies seek greater visibility into their data processes. This growing demand has found a natural synergy with the rise of the data lake. As a result, monitoring data in real time was often an afterthought.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

Towards Data Science

MAY 3, 2023

VDK helps you easily perform complex operations, such as data ingestion and processing from different sources, using SQL or Python. You can use VDK to build data lakes and ingest raw data extracted from different sources, including structured, semi-structured, and unstructured data.

Data Lake 98

Data Lake SQL Data Data Warehouse 98

Cloudera

AUGUST 13, 2021

Every enterprise is trying to collect and analyze data to get better insights into their business. Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake? What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Towards Data Science

DECEMBER 4, 2023

A robust data infrastructure is a must-have to compete in the F1 business. We’ll build a data architecture to support our racing team starting from the three canonical layers : Data Lake, Data Warehouse, and Data Mart. Data Layers can be often combined together, sometimes in a single platform.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Cloudera

JUNE 9, 2022

From origin through all points of consumption both on-prem and in the cloud, all data flows need to be controlled in a simple, secure, universal, scalable, and cost-effective way. controlling distribution while also allowing the freedom and flexibility to deliver the data to different services is more critical than ever. .

Data Collection 94

Data Collection Data Lake Unstructured Data Retail 94

Cloudera

DECEMBER 11, 2020

Without meeting GxP compliance, the Merck KGaA team could not run the enterprise data lake needed to store, curate, or process the data required to inform business decisions. It established a data governance framework within its enterprise data lake. Driving innovation with secure and governed data .

Data Lake 98

Data Lake Government Data Security Unstructured Data 98

Snowflake

JULY 10, 2023

“California Air Resources Board has been exploring processing atmospheric data delivered from four different remote locations via instruments that produce netCDF files. Previously, working with these large and complex files would require a unique set of tools, creating data silos. ” U.S.

Unstructured Data 40

Unstructured Data Python Process Scala 40

Cloudera

NOVEMBER 17, 2022

The Solution: CDP Private Cloud brings a next-generation hybrid architecture with cloud-native benefits to HBL’s data platform. HBL started their data journey in 2019 when data lake initiative was started to consolidate complex data sources and enable the bank to use single version of truth for decision making.

Banking 85

Banking Management Data Lake Professional Services 85

Monte Carlo

MAY 31, 2022

Integrating with it is the holy grail of Spark lineage because it contains all the information needed for how data moves through the data lake and how everything is connected. That makes the DAG the equivalent of a SQL execution plan. It’s a really elegant solution….here’s here’s why it doesn’t work.

Data Lake 52

Data Lake Building Scala Metadata 52

Propel Data

OCTOBER 11, 2022

The main difference between data lakes and data warehouses is data lakes allow unstructured data, but data warehouses need structured data.

Data Lake 40

Data Lake Data Warehouse Unstructured Data Data Analytics 40

Cloudera

MARCH 17, 2023

When implementing a data lakehouse, the table format is a critical piece because it acts as an abstraction layer, making it easy to access all the structured, unstructured data in the lakehouse by any engine or tool, concurrently. Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID.

Education 64

Education Unstructured Data Data Lake Data Warehouse 64

Cloudera

SEPTEMBER 12, 2022

Organizations don’t know what they have anymore and so can’t fully capitalize on it — the majority of data generated goes unused in decision making. And second, for the data that is used, 80% is semi- or unstructured. Both obstacles can be overcome using modern data architectures, specifically data fabric and data lakehouse.

Unstructured Data 75

Unstructured Data Data Lake Data Architecture Data 75

Data Engineering Podcast

NOVEMBER 27, 2022

Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box. images, documents, etc.) images, documents, etc.)

Data Process 100

Data Process Process Metadata Business Intelligence 100

phData: Data Engineering

NOVEMBER 1, 2024

Strong data governance also lays the foundation for better model performance, cost efficiency, and improved data quality, which directly contributes to regulatory compliance and more secure AI systems.

Government 52

Government Data Governance Finance Metadata 52