Towards Data Science

DECEMBER 16, 2024

The unstructured data stack will emerge(Barr) The idea of leveraging unstructured data in production isnt new by any meansbut in the age of AI, unstructured data has taken on a whole newrole. According to a report by IDC only about half of an organizations unstructured data is currently being analyzed.

Unstructured Data 80

Unstructured Data Data Food Data Engineering 80

Cloudera

AUGUST 13, 2021

Every enterprise is trying to collect and analyze data to get better insights into their business. Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

Monte Carlo

MARCH 19, 2025

Monte Carlo and Databricks double-down on their partnership, helping organizations build trusted AI applications by expanding visibility into the data pipelines that fuel the Databricks Data Intelligence Platform. This comprehensive visibility helps teams identify and resolve data issues before they cascade into AI failures.

Unstructured Data 52

Unstructured Data Data Pipeline High Quality Data Banking 52

Snowflake

JANUARY 29, 2025

Snowflake Cortex AI Snowflake Cortex AI is a suite of integrated features and services that include fully-managed LLM inference, fine-tuning, and RAG for structured and unstructured data, to enable customers to quickly analyze unstructured data alongside their structured data, and expedite the building of AI apps.

Unstructured Data 87

Unstructured Data SQL Python Government 87

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Data Engineering Podcast

FEBRUARY 27, 2022

Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow.

Unstructured Data 100

Unstructured Data Cloud Management Metadata 100

Data Engineering Podcast

JUNE 26, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Monte Carlo

MARCH 28, 2025

Failures can be boiled down into one of four root causes: Data First, you have the data feeding your modern data and AI platform. At its most basic, AI is a data product. From model training to the RAG pipelines, data is the heart of the AIand any data + AI quality strategy needs to start here first.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

Data Engineering Podcast

JUNE 19, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. A few highlights from the report Unstructured data goes mainstream. AI-driven code development is going mainstream now.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Data Engineering Weekly

JANUARY 12, 2025

With Astro, you can build, run, and observe your data pipelines in one place, ensuring your mission critical data is delivered on time. Generative AI demands the processing of vast amounts of diverse, unstructured data (e.g.,

Pipeline-centric 57

Pipeline-centric Data Engineering Data Engineer Engineering 57

Snowflake

NOVEMBER 1, 2023

They can also use and leverage Snowflake’s unified governance framework to seamlessly secure and manage access to their data. Cost-effective LLM-based models that are great for working with unstructured data: Answer Extraction (in private preview): Extract information from your unstructured data.

Unstructured Data 123

Unstructured Data SQL Python Accessible 123

Snowflake

JULY 10, 2023

Previously, working with these large and complex files would require a unique set of tools, creating data silos. Now, with unstructured data processing natively supported in Snowflake, we can process netCDF file types, thereby unifying our data pipeline. Mike Tuck, Air Pollution Specialist Why unstructured data?

Unstructured Data 40

Unstructured Data Python Process Scala 40

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Snowflake

JUNE 13, 2024

Bringing in batch and streaming data efficiently and cost-effectively Ingest and transform batch or streaming data in <10 seconds: Use COPY for batch ingestion, Snowpipe to auto-ingest files, or bring in row-set data with single-digit latency using Snowpipe Streaming.

Data Ingestion 107

Data Ingestion MySQL PostgreSQL Data Pipeline 107

Data Engineering Weekly

SEPTEMBER 18, 2024

Decoupling of Storage and Compute : Data lakes allow observability tools to run alongside core data pipelines without competing for resources by separating storage from compute resources. Organizations can track, troubleshoot, and optimize their data pipelines in real-time, ensuring smoother operations and better insights.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. It is a good reminder to the data industry that we need to solve the fundamentals of data engineering to utilize AI better.

Data Engineering 102

Data Engineering Data Engineer Engineering Data Pipeline 102

Monte Carlo

FEBRUARY 12, 2024

Today, this first-party data mostly lives in two types of data repositories. If it is structured data then it’s often stored in a table within a modern database, data warehouse or lakehouse. If it’s unstructured data, then it’s often stored as a vector in a namespace within a vector database.

Database 72

Database Unstructured Data Data Pipeline Metadata 72

Snowflake

FEBRUARY 13, 2025

Lastly, companies have historically collaborated using inefficient and legacy technologies requiring file retrieval from FTP servers, API scraping and complex data pipelines. These processes were costly and time-consuming and also introduced governance and security risks, as once data is moved, customers lose all control.

Management 76

Management Government Cloud Unstructured Data 76

Data Engineering Weekly

OCTOBER 11, 2024

Centralized factories and monolithic data systems became too rigid and expensive to scale, unable to cope with the increasing complexity of manufacturing and the explosion of diverse, unstructured data in the digital age. However, the modern data stack presents challenges like manufacturing's global supply chains.

Manufacturing 109

Manufacturing Transportation Data Warehouse Unstructured Data 109

Cloudera

JUNE 9, 2022

In this second installment of the Universal Data Distribution blog series, we will discuss a few different data distribution use cases and deep dive into one of them. . Data Lakehouse and Cloud Warehouse Ingest : CDF-PC modernizes customer data pipelines with a single tool that works with any data lakehouse or warehouse.

Data Collection 94

Data Collection Data Lake Unstructured Data Retail 94

DataKitchen

APRIL 13, 2021

Airflow — An open-source platform to programmatically author, schedule, and monitor data pipelines. DBT (Data Build Tool) — A command-line tool that enables data analysts and engineers to transform data in their warehouse more effectively. Soda Data Monitoring — Soda tells you which data is worth fixing.

Consulting 120

Consulting Machine Learning Data Science Data Pipeline 120

Data Engineering Podcast

JUNE 12, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Cloudera

OCTOBER 29, 2020

From our release of advanced production machine learning features in Cloudera Machine Learning, to releasing CDP Data Engineering for accelerating data pipeline curation and automation; our mission has been to constantly innovate at the leading edge of enterprise data and analytics.

Machine Learning 111

Machine Learning Data Warehouse Unstructured Data Government 111

Snowflake

APRIL 8, 2024

Many entries also used Snowpark , taking advantage of the ability to work in the code they prefer to develop data pipelines, ML models and apps, then execute in Snowflake. It deploys gen AI components as containers on Snowpark Container Services, close to the customer’s data.

Pipeline-centric 131

Pipeline-centric Food Healthcare Unstructured Data 131

Towards Data Science

APRIL 6, 2023

Alternatively, end-to-end tests, which assess a full system, stretching across repos and services, get overwhelmed by the cross-team complexity of dynamic data pipelines. Unit tests and end-to-end testing are necessary but insufficient to ensure high data quality in organizations with complex data needs and complex tables.

Unstructured Data 98

Unstructured Data Metadata Data Coding 98

Data Engineering Podcast

NOVEMBER 27, 2022

Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. images, documents, etc.)

Data Process 100

Data Process Process Metadata Business Intelligence 100

Data Engineering Podcast

FEBRUARY 24, 2019

What is involved in building a data pipeline and production infrastructure for a deep learning product? How does that differ from other types of analytics projects such as data warehousing or traditional ML? What is involved in building a data pipeline and production infrastructure for a deep learning product?

Deep Learning 100

Deep Learning Data Engineer Data Engineering Engineering 100

Towards Data Science

DECEMBER 4, 2023

We’ll build a data architecture to support our racing team starting from the three canonical layers : Data Lake, Data Warehouse, and Data Mart. Data Lake A data lake would serve as a repository for raw and unstructured data generated from various sources within the Formula 1 ecosystem: telemetry data from the cars (e.g.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Monte Carlo

NOVEMBER 9, 2023

We *know* what we’re putting in (raw, often unstructured data) and we *know* what we’re getting out, but we don’t know how it got there. RAG also affords teams a level of transparency since you know the source of the data that you’re piping into the model to generate new responses. Take GPT-4 for example. While GPT-4 blew GPT 3.5

Unstructured Data 86

Unstructured Data Database Data Pipeline Architecture 86

Snowflake

SEPTEMBER 7, 2023

Sherif Nada, Founding Member & Engineering Manager, Airbyte “External Access in Snowpark is one of the most awaited features for our internal data engineering team at Snowflake. Snowpark External Access is leveraged to build a Ingest and Reverse ETL data pipeline for production workload.

Amazon Web Services 132

Amazon Web Services AWS Government Python 132

Data Engineering Podcast

JULY 31, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. The only thing worse than having bad data is not knowing that you have it.

IT 100

IT Metadata MongoDB MySQL 100

Striim

MARCH 4, 2025

Reimagine Data Governance with Sentinel and Sherlock: Striims AI Agents Striim 5.0 introduces Sentinel and Sherlock, which redefine real-time data governance by seamlessly integrating advanced AI capabilities into your data pipelines. These intelligent agents ensure robust security without sacrificing system performance.

Data Governance 40

Data Governance Government Healthcare NoSQL 40

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Unstructured Data 52

Unstructured Data SQL Data Pipeline Data Validation 52

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is the role of a Data Engineer? Data scientists and data Analysts depend on data engineers to build these data pipelines.

Data Engineer 98

Data Engineer Data Engineering Engineering Hadoop 98