databricks

JUNE 5, 2024

Image datasets are crucial in. In today's data-driven world, the fusion of visual assets and analytical capabilities unlocks a realm of untapped potential.

Datasets 83

Datasets Data 83

Engineering at Meta

JANUARY 22, 2025

In this blog, we will delve into an early stage in PAI implementation: data lineage. This took Meta multiple years to complete across our millions of disparate data assets, and well cover each of these more deeply in future blog posts: Inventorying involves collecting various code and data assets (e.g.,

Data Warehouse 110

Data Warehouse SQL Programming Language Data 110

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. A large international scientist collaboration released The Well : 2 massive datasets from physics simulation (15TB) to astronomical scientific data (100TB). They aim produce the same innovation as ImageNet produced for image recognition.

Data 130

Data Data Warehouse Coding Programming Language 130

Data Engineering Weekly

MARCH 2, 2025

I found the blog to be a fresh take on the skill in demand by layoff datasets. The blog provides an excellent analysis of smallpond compared to Spark and Daft. The blog provides an excellent analysis of smallpond compared to Spark and Daft. Whether you use Datasets already or want to get started, we've got you covered!

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

ThoughtSpot

APRIL 22, 2025

Level 2: Understanding your dataset To find connected insights in your business data, you need to first understand what data is contained in the dataset. Spotter quickly translates your datasets into business-friendly terminology so business users can confidently explore their data through natural language conversations.

BI 59

BI Datasets Business Intelligence Raw Data 59

KDnuggets

AUGUST 24, 2022

This blog post introduces seven techniques that are commonly applied in domains like intrusion detection or real-time bidding, because the datasets are often extremely imbalanced.

Datasets 160

Datasets Data Machine Learning 160

Data Engineering Weekly

NOVEMBER 24, 2024

The blog highlights how moving from 6-character base-64 to 20-digit base-2 file distribution brings more distribution in S3 and reduces request failures. The blog is a good summary of how to use Snowflake QUERY_TAG to measure and monitor query performance. The blog post made me curious to understand DataFusion's internals.

Data Engineering 58

Data Engineering Data Engineer Engineering Insurance 58

KDnuggets

NOVEMBER 30, 2023

The blog discusses five platforms designed for data scientists with specialized capabilities in managing large datasets, models, workflows, and collaboration beyond what GitHub offers.

Data Science 147

Data Science Project Datasets Data 147

Uber Engineering

AUGUST 5, 2021

Uber delivers efficient and reliable transportation across the global marketplace, which is powered by hundreds of services, machine learning models, and tens of thousands of datasets.

Transportation 145

Transportation Datasets Machine Learning Data 145

Pinterest Engineering

APRIL 4, 2025

In this blog, we will go through the technical design and share some offline and online results for our LLM-based search relevance pipeline. Pin Text Representations Pins on Pinterest are rich multimedia entities that feature images, videos, and other contents, often linked to external webpages or blogs.

Machine Learning 56

Machine Learning Metadata Architecture Datasets 56

Data Engineering Weekly

MARCH 16, 2025

link] AWS: An introduction to preparing your own dataset for LLM training Everything in AI eventually comes down to the quality and completeness of your internal data. The blog narrates how Apache Arrow offers better data serialization efficiency and avoids design pitfalls from the past. years of manual effort!!!

Data Engineering 58

Data Engineering Data Engineer Engineering Amazon Web Services 58

Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. To try and predict this, an extensive dataset including anonymised details on the individual loanee and their historical credit history are included. Get the Dataset. Introduction.

Machine Learning 137

Machine Learning Data Science Datasets Raw Data 137

Picnic Engineering

APRIL 7, 2025

A little over a year ago, we shared a blog post about our journey to enhance customers meal planning experience with personalized recipe recommendations. The approach struggled with scalability , making it difficult to handle large datasets efficiently. These connections form a dataset of interactions.

Datasets 52

Datasets Systems Architecture Machine Learning 52

Team Data Science

MARCH 15, 2020

I am taking you through my recent experience to find a dataset for my project. I defined a few sources in my earlier blog post, which will give a sneak peek of techniques to extract industries. Criteria Define a simple layout to your dataset with elements like size, type of columns, format.

Insurance 130

Insurance Datasets Banking Finance 130

Monte Carlo

NOVEMBER 26, 2024

Synthetic data works by leveraging models to create artificial datasets that reflect what someone might find organically (in some alternate reality where more data actually exists), and then using that new data to train their own models. But is synthetic data a long-term solution? Probably not.

Data Engineering 105

Data Engineering Data Engineer Engineering Unstructured Data 105

Netflix Tech

FEBRUARY 14, 2025

In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. Architecture Overview The first pivotal step in managing impressions begins with the creation of a Source-of-Truth (SOT) dataset.

Kafka 66

Kafka Datasets Metadata Utilities 66

DataKitchen

NOVEMBER 5, 2024

The Medallion architecture is a framework that allows data engineers to build organized and analysis-ready datasets in a lakehouse environment. For instance, suppose a new dataset from an IoT device is meant to be ingested daily into the Bronze layer. How do you ensure data quality in every layer?

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

KDnuggets

JULY 5, 2024

In this blog, we will define Pandas and provide an example of how you can vectorize your Python code to optimize dataset analysis using Pandas to speed up your code over 300x times faster.

Python 121

Python Datasets Coding 121

Team Data Science

NOVEMBER 28, 2020

During our first week, Andreas helped us navigate the industry that interests each of us, and this is how we picked a dataset to work on for the next following weeks. I ended up picking yelp dataset , which aligns with my interests in analyzing user behaviors. Stay tuned! Cheers, Liuna Say hello on LinkedIn: [link]

Engineering 130

Engineering Datasets Data Data Engineering 130

Edureka

JANUARY 2, 2025

This blog will walk you through SUMX Power BI Functions , one of these traditional and significant functions. Additionally, it manages sizable datasets without causing Power BI to crash or perform less quickly. The purpose of the Power BI SUMX function was to perform calculations across a table or dataset row by row.

BI 52

BI Datasets Business Intelligence Data Analysis 52

Cloudyard

DECEMBER 24, 2024

In this blog, well explore Building an ETL Pipeline with Snowpark by simulating a scenario where commerce data flows through distinct data layersRAW, SILVER, and GOLDEN.These tables form the foundation for insightful analytics and robust business intelligence. Built clean, enriched datasets in the SILVER layer.

Building 52

Building Raw Data Scala Business Intelligence 52

Pinterest Engineering

APRIL 7, 2025

In this blog post, well discuss our experiences in identifying the challenges associated with EC2 network throttling. For these use cases, typically datasets are generated offline in batch jobs and get bulk uploaded from S3 to the database running on EC2. 4xl with up to 12.5

AWS 57

AWS Bytes Database Data Ingestion 57

WeCloudData

FEBRUARY 17, 2025

In the first blog of the data wrangling series, we introduced the basics of data wrangling using Python. We work on handling missing values, removing special characters, and dropping unnecessary columns to prepare our dataset for further analysis. Welcome back to our Data Wrangling with Python series!

Python 52

Python Datasets Data Data Engineering 52

Data Engineering Weekly

APRIL 6, 2025

The Grab blog delights me since I have tried to do this many times. link] Duolingo: How we built a robust ecosystem for dataset development Duolingo shares how it reimagined data modeling through the lens of software engineering, treating modeled datasets like APIs to enhance consistency, reliability, and developer experience.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. What is Change Data Capture? or its affiliates.

Kafka 50

Kafka MySQL Database Software Engineer 50

Ascend.io

OCTOBER 28, 2024

In this blog post, we’ll explore fundamental concepts, intermediate strategies, and cutting-edge techniques that are shaping the future of data engineering. Filling in missing values could involve leveraging other company data sources or even third-party datasets.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 113

Data Science Cloud Hadoop Metadata 113

DataKitchen

FEBRUARY 20, 2025

Now With Actionable, Automatic, Data Quality Dashboards Imagine a tool that can point at any dataset, learn from your data, screen for typical data quality issues, and then automatically generate and perform powerful tests, analyzing and scoring your data to pinpoint issues before they snowball. Announcing DataOps Data Quality TestGen 3.0:

Datasets 73

Datasets Metadata Data Government 73

databricks

MARCH 26, 2023

In today's business landscape, secure and cost-effective data sharing is more critical than ever for organizations looking to optimize their internal and external.

Datasets 98

Datasets Data 98

databricks

SEPTEMBER 24, 2024

This blog post explores how to create a Genie space using a World of Warcraft dataset, enabling users to interactively query data and gain insights like a data analyst. Unlock the potential of your data with Databricks' AI/BI Genie spaces!

BI 105

BI Datasets Data Analysis Entertainment 105

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. In this blog, we will discuss: What is the Open Table format (OTF)? These formats are transforming how organizations manage large datasets. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Weekly

FEBRUARY 9, 2025

MoEs necessitate less compute for pre-training compared to dense models, facilitating the scaling of model and dataset size within similar computational budgets. I found the product blog from QuantumBlack gives a view of data quality in unstructured data.

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Pinterest Engineering

SEPTEMBER 12, 2023

transformers) became standardized, ML engineers started to show a growing appetite to iterate on datasets. While such dataset iterations can yield significant gains, we observed that only a handful of such experiments were conducted and productionized in the last six months. As model architecture building blocks (e.g.

Data Process 105

Data Process Process Datasets Software Engineer 105

Engineering at Meta

APRIL 28, 2025

Machine learning models : trained on labeled datasets using supervised learning and improved through unsupervised learning to identify patterns and anomalies in unlabeled data. For example, in the data warehouse, it’s represented as a Dataset – an in-code Python class capturing the asset’s schema and metadata.

Metadata 64

Metadata Data Utilities Data Warehouse 64

Uber Engineering

FEBRUARY 23, 2023

In this blog learn how we automated column-level drift detection in batch datasets at Uber scale, reducing the median time to detect issues in critical datasets by 5X. Data quality is of paramount importance at Uber, powering critical decisions and features.

Systems 98

Systems Datasets Data 98

Cloudera

FEBRUARY 22, 2022

The data architecture layer is one such area where growing datasets have pushed the limits of scalability and performance. The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg.

Metadata 110

Metadata Datasets BI SQL 110

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The blog gives an overview of statistical techniques used in drift detection.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98