Christophe Blefari

MARCH 1, 2023

In the ELT, the load is done before the transform part without any alteration of the data leaving the raw data ready to be transformed in the data warehouse. In a simple words dbt sits on top of your raw data to organise all your SQL queries that are defining your data assets.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. It’s the big blueprint we data engineers follow in order to transform raw data into valuable insights. Why Flink instead of Spark?

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Snowflake

NOVEMBER 11, 2024

At Snowflake BUILD , we are introducing powerful new features designed to accelerate building and deploying generative AI applications on enterprise data, while helping you ensure trust and safety. These scalable models can handle millions of records, enabling you to efficiently build high-performing NLP data pipelines.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Cloudera

JANUARY 9, 2025

Executives, data teams, and even end-users understand that AI means more than building models; it means unlocking strategic value. By trimming acronyms and complex naming, were removing barriers so you can quickly find what you need and get to work building intelligence into every decision. Ready to learn more?

Machine Learning 85

Machine Learning Raw Data Government Algorithm 85

Cloudyard

JANUARY 15, 2025

However, Ive taken this a step further, leveraging Snowpark to extend its capabilities and build a complete data extraction process. This blog explores how you can leverage the power of PARSE_DOCUMENT with Snowpark, showcasing a use case to extract, clean, and process data from PDF documents. policy holder name, policy number).

Data Cleanse 52

Data Cleanse Insurance Raw Data Unstructured Data 52

KDnuggets

NOVEMBER 17, 2021

With the customer at its heart, modern augmented BI platforms no longer require scripting/coding skills or the knowledge to build the back-end data models, empowering even laymen to harness the power of raw data. As a user, here are the top AI capabilities that you need to look for in BI software.

BI 134

BI Raw Data Coding Skills Coding 134

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? The Platform Integration Data Store Transformation Orchestration Presentation Transportation Observability Closing What’s changed?

Building 98

Building Transportation Data Lake Metadata 98

AltexSoft

MAY 12, 2022

The same relates to those who buy annotated sound collections from data providers. But if you have only raw data meaning recordings saved in one of the audio file formats you need to get them ready for machine learning. Audio data labeling. Building an app for snore and teeth grinding detection.

Machine Learning 122

Machine Learning Building Deep Learning Healthcare 122

Lyft Engineering

APRIL 21, 2023

Building a large scale unsupervised model anomaly detection system — Part 1 Distributed Profiling of Model Inference Logs By Anindya Saha , Han Wang , Rajeev Prabhakar Introduction LyftLearn is Lyft’s ML Platform. We instrument all inference requests, sample and store a certain percentage of model inference requests and emitted predictions.

Systems 109

Systems Building Machine Learning Raw Data 109

Ascend.io

OCTOBER 28, 2024

What is Data Transformation? Data transformation is the process of converting raw data into a usable format to generate insights. It involves cleaning, normalizing, validating, and enriching data, ensuring that it is consistent and ready for analysis.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

DataKitchen

DECEMBER 6, 2024

We covered how Data Quality Testing, Observability, and Scorecards turn data quality into a dynamic process, helping you build accuracy, consistency, and trust at each layerBronze, Silver, and Gold.

Architecture 52

Architecture Raw Data High Quality Data Data Validation 52

Snowflake

JANUARY 29, 2024

Bring your raw Google Analytics data to Snowflake with just a few clicks The Snowflake Connector for Google Analytics makes it a breeze to get your Google Analytics data, either aggregated data or raw data, into your Snowflake account. Here’s a quick guide to get started: 1. But don’t just take it from us.

Raw Data 126

Raw Data Aggregated Data Cloud Data 126

Towards Data Science

MARCH 9, 2023

Code implementations for ML pipelines: from raw data to predictions Photo by Rodion Kutsaiev on Unsplash Real-life machine learning involves a series of tasks to prepare the data before the magic predictions take place.

Machine Learning 98

Machine Learning Building Datasets Big Data 98

The Pragmatic Engineer

JANUARY 12, 2023

In this week’s The Scoop, I analyzed this information and dissected it, going well beyond the raw data. Here are a few details from the data points, focusing on software engineering compensation. Every employee could examine a PDF with compensation details for every role at the company.

Software Engineering 219

Software Engineering Software Engineer Engineering Raw Data 219

The Pragmatic Engineer

JUNE 13, 2023

Unlike Uber, Agoda does not make use of public cloud providers, having decided to build out its own private cloud, instead. This group doesn’t include the software layer for infrastructure, which is a software team that builds the orchestration platform (Fleet) upon Kubernetes. In some cases this makes sense.

Cloud 251

Cloud Database Utilities BI 251

Towards Data Science

FEBRUARY 19, 2024

Image from Unsplash Building a Semantic Book Search: Scale an Embedding Pipeline with Apache Spark and AWS EMR Serverless Using OpenAI’s Clip model to support natural language search on a collection of 70k book covers In a previous post I did a little PoC to see if I could use OpenAI’s Clip model to build a semantic book search.

AWS 90

AWS Building Python Bytes 90

Towards Data Science

FEBRUARY 6, 2024

As you do not want to start your development with uncertainty, you decide to go for the operational raw data directly. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Does it sound familiar?

Systems 98

Systems Raw Data Metadata Data Cleanse 98

Lyft Engineering

APRIL 25, 2023

Building a large scale unsupervised model anomaly detection system — Part 2 Building ML Models with Observability at Scale By Rajeev Prabhakar , Han Wang , Anindya Saha Photo by Octavian Rosca on Unsplash In our previous blog we discussed the different challenges we faced for model monitoring and our strategy for addressing some of these problems.

Systems 81

Systems Building Machine Learning Raw Data 81

Edureka

APRIL 14, 2025

Understanding the Tools One platform is designed primarily for business intelligence, offering intuitive ways to connect to various data sources, build interactive dashboards, and share insights. Its purpose is to simplify data exploration for users across skill levels. What is Power BI?

BI 40

BI Business Intelligence Raw Data Retail 40

Engineering at Meta

JANUARY 21, 2025

This is one of the biggest advantages of building profilers using eBPF: complex and customized actions taken at sample time. Strobelight also delays symbolization until after profiling and stores raw data to disk to prevent memory thrash on the host. To add to that enchilada (hungry yet?),

Technology 114

Technology Metadata Utilities Engineering 114

Precisely

NOVEMBER 20, 2023

We work with organizations around the globe that have diverse needs but can only achieve their objectives with expertly curated data sets containing thousands of different attributes. Practitioners can rely on consistent data to extract meaningful features contributing to model performance. Clean data reduces the need for data prep.

Raw Data 121

Raw Data Insurance Data Portfolio 121

Hevo

FEBRUARY 15, 2023

After the hustle and bustle of extracting data from multiple sources, you have finally loaded all your data to a single source of truth like the Snowflake data warehouse. However, data modeling is still challenging and critical for transforming your raw data into any analysis-ready form to get insights.

Data Pipeline 59

Data Pipeline Building Raw Data Data Warehouse 59

Snowflake

NOVEMBER 27, 2023

Welcome to Snowflake’s Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. Traveling over hard ground on the way to building something important is what inspires me. Hum is harnessing frontier AI to transform content and audience data into actionable insights and personalized experiences.

Raw Data 124

Raw Data Relational Database Consulting Architecture 124

Maxime Beauchemin

AUGUST 28, 2017

The data warehouse needs to reflect the business, and the business should have clarity on how it thinks about analytics. Conflicting nomenclature and inconsistent data across different namespaces, or “ data marts ” are problematic. Data engineers are many degrees removed from those who are “moving the needle”.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineering 174

DataKitchen

AUGUST 9, 2021

The data industry has a wide variety of approaches and philosophies for managing data: Inman data factory, Kimball methodology, s tar schema , or the data vault pattern, which can be a great way to store and organize raw data, and more. Data mesh does not replace or require any of these.

Pharmaceutical 98

Pharmaceutical Raw Data Data Data Lake 98

Christophe Blefari

APRIL 21, 2023

If a model do not respect a contract it will not build. In dbt vocabulary build means run + other things. Building a ChatGPT Plugin for Medium. Fast News ⚡️ Building a Flink self-serve platform on Kubernetes at scale — Instacart engineering team migrated from Flink on EMR to Flink on Kubernetes.

Raw Data 130

Raw Data Data SQL Datasets 130

Precisely

SEPTEMBER 25, 2023

Read our eBook Validation and Enrichment: Harnessing Insights from Raw Data In this ebook, we delve into the crucial data validation and enrichment process, uncovering the challenges organizations face and presenting solutions to simplify and enhance these processes.

Data Validation 100

Data Validation Process Raw Data Data Cleanse 100

Towards Data Science

NOVEMBER 16, 2023

Now that you have learned what batch data processing is, let’s move on to the next step: creating and managing batch processing pipelines in VDK. 2 Creating and Managing Batch Processing Pipelines in VDK VDK adopts a component-based approach, enabling you to build data processing pipelines quickly. link] Summary Congratulations!

Data Process 98

Data Process Process Raw Data Data 98

Monte Carlo

JANUARY 29, 2025

We talked about their plans for GenAI and the challenges theyve encountered as they incorporate large language models (LLMs) into their data products while prioritizing consistency and reliability. The company uses a medallion architecture, where data flows from raw (bronze) to standardized (silver) to aggregated (gold) layers.

Media 52

Media Data Science Datasets Raw Data 52

Cloudera

MAY 19, 2021

RAPIDS on the Cloudera Data Platform comes pre-configured with all the necessary libraries and dependencies to bring the power of RAPIDS to your projects. RAPIDS brings the power of GPU compute to standard Data Science operations, be it exploratory data analysis, feature engineering or model building. Data Ingestion.

Machine Learning 137

Machine Learning Data Science Datasets Raw Data 137

Confluent

OCTOBER 16, 2019

There’s also some static reference data that is published on web pages. ?After Wrangling the data. With the raw data in Kafka, we can now start to process it. Since we’re using Kafka, we are working on streams of data. After we scrape these manually, they are produced directly into a Kafka topic.

Kafka 19

Kafka Building Data Coding 19

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Snowflake

APRIL 7, 2023

This was the first year that startups had the chance to build with our Native Applications Framework (currently in private preview), and we were thrilled to see the number of entries that included a native app. It transforms multiple financial and operational systems’ raw data into a common, friendly data model that people can understand.

Raw Data 111

Raw Data Portfolio Building SQL 111

Cloudera

NOVEMBER 17, 2020

The second Applied Machine Learning Prototype that was made available is for building a fraud detection model. . These are prototypes that will help you build a fully working machine learning example in CML. The Templates will include source data and walk through various steps: Ingest data into a useful place in CDP (e.g.

Deep Learning 115

Deep Learning Machine Learning Raw Data Data Ingestion 115

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 105

Engineering Raw Data Data Science Machine Learning 105

Monte Carlo

NOVEMBER 21, 2024

Want to run SQL queries on your structured data while also keeping raw files for your data scientists to play with? The data lakehouse has got you covered! Data typically flows through three stages: Bronze : Raw data lands here first, preserved in its original form.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

KDnuggets

NOVEMBER 17, 2021

With the customer at its heart, modern augmented BI platforms no longer require scripting/coding skills or the knowledge to build the back-end data models, empowering even laymen to harness the power of raw data. As a user, here are the top AI capabilities that you need to look for in BI software.

BI 83

BI Raw Data Coding Skills Coding 83