Waitingforcode

JANUARY 15, 2025

A few months ago I wrote a blog post about event skew and how dangerous it is for a stateful streaming job. Since it was a high-level explanation, I didn't cover Apache Spark Structured Streaming deeply at that moment.

IT 130

IT 130

KDnuggets

JUNE 9, 2025

Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid Ali Awan ( @1abidaliawan ) is a certified data scientist professional who loves building machine learning models.

Telecommunication 131

Telecommunication Data Science Machine Learning Python 131

databricks

JUNE 11, 2025

Events Data + AI Summit Data + AI World Tour Data Intelligence Days Event Calendar Blog and Podcasts Databricks Blog Explore news, product announcements, and more Databricks Mosaic Research Blog Discover the latest in our Gen AI research Data Brew Podcast Let’s talk data! REGISTER Ready to get started?

Entertainment 81

Entertainment Data Lake Manufacturing Consulting 81

databricks

JUNE 11, 2025

Events Data + AI Summit Data + AI World Tour Data Intelligence Days Event Calendar Blog and Podcasts Databricks Blog Explore news, product announcements, and more Databricks Mosaic Research Blog Discover the latest in our Gen AI research Data Brew Podcast Let’s talk data! For more details, see the Agent Bricks deep dive blog.

Entertainment 75

Entertainment Manufacturing Consulting Retail 75

Engineering at Meta

JANUARY 22, 2025

In this blog, we will delve into an early stage in PAI implementation: data lineage. This took Meta multiple years to complete across our millions of disparate data assets, and well cover each of these more deeply in future blog posts: Inventorying involves collecting various code and data assets (e.g.,

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

KDnuggets

JUNE 13, 2025

Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid Ali Awan ( @1abidaliawan ) is a certified data scientist professional who loves building machine learning models.

Telecommunication 119

Telecommunication Data Science Machine Learning Python 119

Yelp Engineering

JANUARY 21, 2025

This blog post delves into our journey of optimizing training time using TensorFlow and Horovod, along with the development of ArrowStreamServer, our in-house library for low-latency data streaming and serving. These models handle large tabular datasets with small parameter spaces, requiring innovative data solutions.

Datasets 104

Datasets Architecture Data Solutions Data 104

Cloudera

NOVEMBER 13, 2024

The post Octopai Acquisition Enhances Metadata Management to Trust Data Across Entire Data Estate appeared first on Cloudera Blog.

Metadata 90

Metadata Management Data Governance Government 90

Confessions of a Data Guy

MARCH 4, 2025

The blog post reviews an Apache Incubating project called Apache XTable, which aims to provide cross-format interoperability among Delta Lake, Apache Hudi, and Apache Iceberg. Below is a concise breakdown from some time I spend playing around this this new tool and some technical observations: 1. What is Apache XTable?

Project 100

Project Data IT Big Data 100

KDnuggets

JUNE 11, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Bridging the Gap: New Datasets Push Recommender Research Toward Real-World Scale Publicly available (..)

Datasets 125

Datasets Metadata Data Science Machine Learning 125

Cloudera

NOVEMBER 13, 2024

The post Introducing Cloudera Fine Tuning Studio for Training, Evaluating, and Deploying LLMs with Cloudera AI appeared first on Cloudera Blog. Get started today on your fine tuning needs, and Cloudera AI’s team is ready to assist in fulfilling your enterprise’s vision for AI-ready applications to become a reality.

Datasets 87

Datasets Machine Learning Coding AWS 87

Snowflake

JANUARY 15, 2025

Over the last three geospatial-centric blog posts, weve covered the basics of what geospatial data is, how it works in the broader world of data and how it specifically works in Snowflake based on our native support for GEOGRAPHY , GEOMETRY and H3.

Transportation 97

Transportation BI Database-centric Metadata 97

Cloudera

DECEMBER 4, 2024

Read this blog to learn more about how Amazon EMR seamlessly integrates with Cloudera’s lakehouse for secure data sharing and interoperability powered by Iceberg REST Catalog. The post Cloudera announces ‘Interoperability Ecosystem’ with founding members AWS and Snowflake appeared first on Cloudera Blog.

AWS 89

AWS Raw Data Relational Database Government 89

Snowflake

NOVEMBER 26, 2024

This blog post is the second in a three-part series on migrations. That’s why we’ve collected these migration success stories to help you get started on your migration to Snowflake. Today we’re focusing on customers who migrated from a cloud data warehouse to Snowflake and some of the benefits they saw.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Data Lake 105

Scott Logic

JUNE 10, 2025

Useful links for this episode Data engineering posts on the Scott Logic blog Tom Fairbairn’s posts on the Solace blog Event-Driven Integration posts on the Solace blog Subscribe to the podcast Apple Podcasts Spotify

Architecture 52

Architecture Data Integration Retail Finance 52

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables.

Data 130

Data Data Warehouse Programming Language BI 130

KDnuggets

JUNE 10, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Integrating DuckDB & Python: An Analytics Guide Learn how to run lightning-fast SQL queries on (..)

Python 114

Python Data Science SQL Machine Learning 114

ProjectPro

JUNE 6, 2025

This blog introduces the concept of time series forecasting models in the most detailed form. The blog's last two parts cover various use cases of these models and projects related to time series analysis and forecasting problems. This blog will explore these use cases with practical time series forecasting model examples.

Deep Learning 66

Deep Learning Python Machine Learning Datasets 66

Cloudera

OCTOBER 10, 2024

The post Cloudera Lakehouse Optimizer Makes it Easier Than Ever to Deliver High-Performance Iceberg Tables appeared first on Cloudera Blog. Go sign up for our 5-day trial here to see for yourself.

IT 84

IT Data Lake Metadata Data Warehouse 84

Pinterest Engineering

APRIL 4, 2025

In this blog, we will go through the technical design and share some offline and online results for our LLM-based search relevance pipeline. Pin Text Representations Pins on Pinterest are rich multimedia entities that feature images, videos, and other contents, often linked to external webpages or blogs.

Machine Learning 65

Machine Learning Metadata Architecture Datasets 65

databricks

JUNE 4, 2025

Events Data + AI Summit Data + AI World Tour Data Intelligence Days Event Calendar Blog and Podcasts Databricks Blog Explore news, product announcements, and more Databricks Mosaic Research Blog Discover the latest in our Gen AI research Data Brew Podcast Let’s talk data! REGISTER Ready to get started?

AWS 52

AWS Entertainment Manufacturing Media 52

Cloudera

DECEMBER 4, 2024

Delivers Enhanced Efficiency and Adaptability appeared first on Cloudera Blog. Learn More: To explore the new capabilities of Cloudera DataFlow 2.9 and discover how it can transform your data pipelines, watch this video. link] [link] The post Fueling the Future of GenAI with NiFi: Cloudera DataFlow 2.9

Data Pipeline 93

Data Pipeline Data Ingestion Data Preparation Architecture 93

Data Engineering Weekly

MAY 25, 2025

The blog is an excellent compilation of types of query engines on top of the lakehouse, its internal architecture, and benchmarking against various categories. I think the market is wide open for more innovations, as Onehouse announces a compute runtime named Quanton. link] Gunnar Morling: What If We Could Rebuild Kafka From Scratch?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

KDnuggets

JUNE 11, 2025

Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Masters degree in technology management and a bachelors degree in telecommunication engineering.

Telecommunication 78

Telecommunication Data Science Machine Learning Python 78

Snowflake

JANUARY 16, 2025

You can learn more in our SwiftKV research blog post. Because SwiftKV is fully open source, you can also deploy it on your own with model checkpoints on Hugging Face and optimized inference on vLLM.

Algorithm 97

Algorithm Data Analysis Process Building 97

ProjectPro

JUNE 6, 2025

This blog offers an Agentic AI learning path, explaining the core components behind AI Agents. If you’ve ever wondered how these intelligent systems work or wanted to build one, this blog is your starting point. Read more about AI agents in our latest blog: AI Agents: The New Human-Like Heroes of AI.

Deep Learning 80

Deep Learning Algorithm Machine Learning Banking 80

KDnuggets

JUNE 9, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter 7 Cool Python Projects to Automate the Boring Stuff Get more done in less time with these 7 beginner-friendly (..)

Python 106

Python Project Media Data Science 106

Data Engineering Weekly

JUNE 1, 2025

The blog narrates adopting a hybrid approach with AWS Sagemaker integration and Chalk feature store. link] EloElo: Building EloElo’s Data Platform - Our 2-year Journey to Batch + Real-Time Lakehouse on Open Source stack The blog captures the emerging lakehouse architecture, combining CDC pipelines and event sourcing systems.

Data Engineering 70

Data Engineering Data Engineer Engineering Relational Database 70

Snowflake

NOVEMBER 12, 2024

Snowpark Container Services (now generally available on Azure as well as AWS) gives developers the flexibility to easily and securely deploy complex functionality — from custom frontends and large-scale ML training and inference to open source and homegrown models — all securely within Snowflake (learn more in this blog post ).

AWS 97

AWS Cloud Programming Language High Quality Data 97

KDnuggets

JUNE 11, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Why You Need RAG to Stay Relevant as a Data Scientist How retrieval-augmented generation (RAG) reduces (..)

Data Science 99

Data Science Machine Learning Python SQL 99

Pinterest Engineering

JANUARY 31, 2025

Conclusion and FutureWorks In this blog, we presented our work in building our learned retrieval system across different surfaces in Pinterest. It has the top user coverage and top three save rates. Since launched, it has helped deprecate two other candidate generators with huge overall site engagement wins.

Systems 75

Systems Metadata Machine Learning Architecture 75

ProjectPro

JUNE 6, 2025

We will cover all such details in this blog. Using familiar SQL as Athena queries on raw data stored in S3 is easy; that is an important point, and you will explore real-world examples related to this in the latter part of the blog. What is AWS Athena?, How to write an AWS Athena query? Table of Contents What is AWS Athena?

AWS 67

AWS Big Data SQL Raw Data 67

Pinterest Engineering

APRIL 7, 2025

In this blog post, well discuss our experiences in identifying the challenges associated with EC2 network throttling. In the remainder of this blog post, well share how we root cause and mitigate the aboveissues. This prompted us to engage with AWS and dive deep into the network performance of our clusters. 4xl with up to 12.5

AWS 66

AWS Bytes Data Ingestion Database 66

Netflix Tech

APRIL 8, 2025

By Cheng Xie , Bryan Shultz , and Christine Xu In a previous blog post , we described how Netflix uses eBPF to capture TCP flow logs at scale for enhanced network insights. In this post, we delve deeper into how Netflix solved a core problem: accurately attributing flow IP addresses to workload identities.

AWS 77

AWS Kafka Programming Cloud 77

Data Engineering Weekly

MAY 4, 2025

The blog highlights how emerging AI tools automate otherwise cognitively intensive manual tasks to bring reliability in software engineering. link] Uber: Fixrleak - Fixing Java Resource Leaks with GenAI Another interesting article from Uber demonstrates how AI significantly accelerates the reliability effects.

Data Engineering 67

Data Engineering Data Engineer Engineering Java 67

The Pragmatic Engineer

OCTOBER 18, 2024

WP Engine is the challenger for the most popular managed WordPress hosting service – generating likely around $400M/year in revenue ( as per Automattic ), versus Automattic’s circa $500M/year, as per Automattic’s CEO, in a now-edited blog post. Automattic raised $980M in venture funding and was valued at $7.5B

Government 357

Government Engineering Project AWS 357

Netflix Tech

JANUARY 2, 2025

However, due to the absence of a control group in these countries, we adopt a synthetic control framework ( blog post ) to estimate the counterfactual scenario. Most UA campaigns are conducted at the country level, meaning that everyone in the targeted countries can see the ads.

Engineering 81

Engineering Entertainment Designing Technology 81