Data Engineering Podcast

OCTOBER 7, 2019

Summary The process of exposing your data through a SQL interface has many possible pathways, each with their own complications and tradeoffs. One of the recent options is Rockset, a serverless platform for fast SQL analytics on semi-structured and structured data. Visit Datacoral.com today to find out more.

Structured Data 100

Structured Data Cloud SQL Programming Language 100

KDnuggets

JUNE 23, 2023

Learn how to use large language models to extract insights from documents for analytics and ML at scale. Join this webinar and live tutorial to learn how to get started.

Structured Data 105

Structured Data Data 105

KDnuggets

JULY 11, 2023

The article highlights various use cases of synthetic data, including generating confidential data, rebalancing imbalanced data, and imputing missing data points. It also provides information on popular synthetic data generation tools such as MOSTLY AI, SDV, and YData.

Structured Data 107

Structured Data Data IT Data Science 107

Monte Carlo

NOVEMBER 26, 2024

Here’s where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024—plus a few predictions of my own. 2025 data engineering trends incoming. Small data is the future of AI (Tomasz) 7. The lines are blurring for analysts and data engineers (Barr) 8. Table of Contents 1.

Data Engineer 105

Data Engineer Data Engineering Engineering Unstructured Data 105

Snowflake

MARCH 26, 2025

In this edition, we talk to Richard Meng, co-founder and CEO of ROE AI , a startup that empowers data teams to extract insights from unstructured, multimodal data including documents, images and web pages using familiar SQL queries. I experienced the thrilling pace of AI data innovation firsthand.

Unstructured Data 98

Unstructured Data SQL Data Data Workflow 98

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

Snowflake

FEBRUARY 18, 2025

Being able to leverage unstructured data is a critical part of an effective data strategy for 2025 and beyond. Even though its such a huge proportion of an enterprises data, many financial services organizations still dont know how to effectively use it. Parse data: What does analyzing unstructured data look like?

Unstructured Data 87

Unstructured Data Insurance Structured Data Government 87

Snowflake

JANUARY 30, 2025

Use cases range from getting immediate insights from unstructured data such as images, documents and videos, to automating routine tasks so you can focus on higher-value work. Gen AI makes this all easy and accessible because anyone in an enterprise can simply interact with data by using natural language.

Unstructured Data 97

Unstructured Data Entertainment Healthcare Telecommunication 97

Towards Data Science

JULY 13, 2023

A step-by-step guide to creating a knowledge graph and exploring its potential to enhance an LLM Continue reading on Towards Data Science »

Structured Data 93

Structured Data Data Science Data IT 93

Towards Data Science

DECEMBER 16, 2024

Agentic AI, small data, and the search for value in the age of the unstructured datastack. Heres where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024plus a few predictions of myown. 2025 data engineering trends incoming. Search: tools that leverage a corpus of data to answer questions 3.

Unstructured Data 80

Unstructured Data Data Food Data Engineer 80

Analytics Vidhya

FEBRUARY 2, 2023

Introduction In this constantly growing era, the volume of data is increasing rapidly, and tons of data points are produced every second. Now, businesses are looking for different types of data storage to store and manage their data effectively.

Data Lake 207

Data Lake Data Warehouse Data Storage Data 207

Snowflake

MARCH 6, 2025

Large language models (LLMs) are transforming how we extract value from this data by running tasks from categorization to summarization and more. While AI has proved that real-time conversations in natural language are possible with LLMs, extracting insights from millions of unstructured data records using these LLMs can be a game changer.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

databricks

JUNE 2, 2024

We are excited to announce a new data type called variant for semi-structured data. Variant provides an order of magnitude performance improvements compared.

Structured Data 136

Structured Data Data Data Engineering Data Engineer 136

Snowflake

NOVEMBER 11, 2024

At Snowflake BUILD , we are introducing powerful new features designed to accelerate building and deploying generative AI applications on enterprise data, while helping you ensure trust and safety. These scalable models can handle millions of records, enabling you to efficiently build high-performing NLP data pipelines.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Simon Späti

MAY 3, 2023

In case you missed Part 1, An Introduction to Data Modeling, make sure to check first, where we discussed the importance of data modeling in data engineering, the history, and the increasing complexity of data. We have also touched upon the significance of understanding the data landscape, its challenges, and much more.

Data Engineer 130

Data Engineer Data Engineering Engineering Structured Data 130

Simon Späti

MAY 3, 2023

In case you missed Part 1, An Introduction to Data Modeling, make sure to check first, where we discussed the importance of data modeling in data engineering, the history, and the increasing complexity of data. We have also touched upon the significance of understanding the data landscape, its challenges, and much more.

Data Engineer 130

Data Engineer Data Engineering Engineering Structured Data 130

Data Engineering Weekly

MARCH 5, 2025

The modern data stack constantly evolves, with new technologies promising to solve age-old problems like scalability, cost, and data silos. It promised to address key pain points: Scaling: Handling ever-increasing data volumes. Speed: Accelerating data insights. Data Silos: Breaking down barriers between data sources.

Hadoop 58

Hadoop Metadata Data Ingestion Data Governance 58

Azure Data Engineering

MAY 15, 2022

When it comes to transforming structured data, (e.g., The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures. The Stored Procedure Activity in Data Factory provides and simple and convenient way to execute Stored Procedures.

SQL 130

SQL Database Structured Data Data 130

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

Data Engineering Weekly

FEBRUARY 9, 2025

link] QuantumBlack: Solving data quality for gen AI applications Unstructured data processing is a top priority for enterprises that want to harness the power of GenAI. It brings challenges in data processing and quality, but what data quality means in unstructured data is a top question for every organization.

Data Engineer 58

Data Engineer Data Engineering Engineering Unstructured Data 58

Monte Carlo

NOVEMBER 21, 2024

Let’s set the scene: your company collects data, and you need to do something useful with it. Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Monte Carlo

MARCH 3, 2025

The real disruption lies with data + AI. In other words, when organizations combine their first-party data with LLMs to unlock unique insights, automate processes, or accelerate specialized workflows. We saw this with software and application observability; data and data observability; and soon data + AI and data + AI observability.

Unstructured Data 52

Unstructured Data Data Cloud Computing Banking 52

Edureka

APRIL 22, 2025

Selecting the appropriate data platform becomes crucial as businesses depend more and more on data to inform their decisions. Although they take quite different approaches, Microsoft Fabric and Snowflake, two of the top players in the current data landscape, both provide strong capabilities. What do you mean by Microsoft Fabric?

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Monte Carlo

OCTOBER 31, 2024

The rise of AI and GenAI has brought about the rise of new questions in the data ecosystem – and new roles. One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Weekly

JANUARY 12, 2025

With Astro, you can build, run, and observe your data pipelines in one place, ensuring your mission critical data is delivered on time. link] Sponsored: Apache Airflow® Best Practices: Running Airflow at Scale The scalability of Airflow is why data teams at companies like Uber, Ford, and LinkedIn choose it to power their data ops.

Pipeline-centric 58

Pipeline-centric Data Engineer Data Engineering Engineering 58

Snowflake

JANUARY 29, 2025

We do not share data with the model provider. To address these issues the DeepSeek team describes how they incorporated cold-start data before RL for enhanced reasoning performance. Governance controls can be implemented consistently across data and AI. To request access during preview please reach out to your sales team.

Unstructured Data 97

Unstructured Data SQL Python Government 97

Christophe Blefari

JANUARY 14, 2024

Back to the usual Data News—with a little delay, I'm sorry. It's a subject close to my heart and I was very happy to share it with you, because I never thought that Data News would become such a big part of my life. I actually cover data engineering and how to put data stuff into production.

Google Cloud 130

Google Cloud Data SQL Data Engineer 130

Analytics Vidhya

FEBRUARY 25, 2023

Introduction A data lake is a centralized and scalable repository storing structured and unstructured data. The need for a data lake arises from the growing volume, variety, and velocity of data companies need to manage and analyze.

Cloud Storage 162

Cloud Storage Data Lake Cloud Unstructured Data 162

Data Engineering Weekly

JULY 14, 2024

link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. Techniques for turning text data and documents into vector embeddings and structured data.

Data Engineer 98

Data Engineer Data Engineering Engineering Unstructured Data 98

Waitingforcode

JANUARY 23, 2025

Even though Apache Spark SQL provides an API for structured data, the framework sometimes behaves unexpectedly. It's the case of an insertInto operation that can even lead to some data quality issues. Let's try to understand in this short article.

SQL 130

SQL Structured Data IT Data 130

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 105

Engineering Raw Data Data Science Machine Learning 105

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Introduction to the Data Mesh Architecture and its Required Capabilities. Components of a Data Mesh.

Architecture 124

Architecture Metadata Kafka Government 124

Data Engineering Podcast

FEBRUARY 9, 2020

Summary Designing the structure for your data warehouse is a complex and challenging process. As businesses deal with a growing number of sources and types of information that they need to integrate, they need a data modeling strategy that provides them with flexibility and speed.

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Cloudyard

JANUARY 15, 2025

Read Time: 2 Minute, 33 Second Snowflakes PARSE_DOCUMENT function revolutionizes how unstructured data, such as PDF files, is processed within the Snowflake ecosystem. Traditionally, this function is used within SQL to extract structured content from documents. Apply advanced data cleansing and transformation logic using Python.

Data Cleanse 52

Data Cleanse Insurance Raw Data Unstructured Data 52

Towards Data Science

JANUARY 16, 2024

My personal take on justifying the existence of Data Mesh A senior stakeholder at one my projects mentioned that they wanted to decentralise their data platform architecture and democratise data across the organisation. When I heard the words ‘decentralised data architecture’, I was left utterly confused at first!

Data Warehouse 95

Data Warehouse Data Architecture Relational Database NoSQL 95

Snowflake

MARCH 14, 2024

As a cohesive ERP solution, SAP is often one of the largest data resources in an organization, containing everything from financial and transactional data to master information about customers, vendors, materials, facilities, planning and even HR. What’s the challenge with unlocking SAP data?

IT 121

IT Data Ingestion Data AWS 121