Waitingforcode
MAY 7, 2025
One of the biggest changes for PySpark has been the DataFrame API. It greatly reduces the JVM-to-PVM communication overhead and improves the performance. However, it also complexities the code. Probably, some of you have already seen, written, or worked with the code like this.
databricks
MAY 7, 2025
As organizations consolidate analytics workloads to Databricks, they often need to adapt traditional data warehouse techniques.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
MAY 5, 2025
A step-by-step guide to securing a FastAPI machine learning applications' endpoints with native authentication and user management.
Jesse Anderson
MAY 5, 2025
I n this episode of Unapologetically Technical, I interview Shane Murray, Field CTO at Monte Carlo Data. Shane shares his compelling journey from studying math and finance in Sydney, Australia, to leading AI strategy at a major data observability company in New York. We explore his early work in choice modeling and pioneering online multivariate experimentation long before A/B testing became mainstream, including fascinating examples from cruise lines, American Express, and even cultural surpris
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
databricks
MAY 5, 2025
Atlassian recently partnered with Databricks to power new data sharing capabilities from Atlassian Analytics, using the Delta Sharing protocol.
Uber Engineering
MAY 5, 2025
Goodbye resource leaks! Learn how the FixrLeak framework leverages GenAI and AST-level analysis to automatically detect and fix resource leaks in large-scale Java applications at Uber.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Engineering at Meta
MAY 5, 2025
Meta and Quantsight have improved key libraries in the Python Ecosystem. There is plenty more to do and we invite the community to help with our efforts. Well look at two key efforts in Pythons packaging ecosystem to make packages faster and easier to use: Unlock performance wins for developers through free-threaded Python where we leverage Python 3.13s support for concurrent programming (made possible by removing the Global Interpreter Lock (GIL)).
Snowflake
MAY 8, 2025
Data providers want their data available to their customers, no matter where in the world or on which cloud service provider the customer is located. However, egress costs can contribute up to 70% of total data transfer costs. Providers have historically had to balance the desire to increase the availability of their data to any relevant Snowflake regions with the need to manage egress costs.
Precisely
MAY 5, 2025
Delivering seamless, personalized experiences for customers across channels continues to be a priority for organizations across industries. To make this goal a reality, they seek out powerful customer communications management (CCM) solutions. However, theres often a debate on whether to build a custom in-house solution or purchase an enterprise-grade platform.
KDnuggets
MAY 5, 2025
Master element-wise operations, comparisons, logic, aggregation, and broadcasting using NumPy ufuncs for high-performance array processing.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Uber Engineering
MAY 5, 2025
Migrating tech stacks at Ubers scale isnt easy. Learn how we migrated our stateless container orchestration platform to Kubernetes and operate it at a scale of 3 million cores with 1.5 million pod launches daily.
Start Data Engineering
MAY 5, 2025
1. Introduction 2. Step-by-step process to solve any SQL interview question 2.1. Define what the input data is and how they are related 2.2. Understand the input table’s grain, foreign keys, and how they relate to each other 2.3. Define the dimensions and metrics required for the output 2.4. Filter/Join/Group by input columns to get the output dimension and metrics 3.
databricks
MAY 5, 2025
Why migrate from Netezza to Databricks? The limitations of traditional enterprise data warehouse (EDW) appliances like Netezza are becoming increasingly apparent.
Edureka
MAY 8, 2025
When you purchase online, do you ever find yourself pondering how your tastes get changed into suggestions for products that are uniquely suited to you? Or how self-driving cars get through very complicated situations with amazing accuracy? These are the ways that data engineering improves our lives in the real world. The field of data engineering turns unstructured data into ideas that can be used to change businesses and our lives.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
InData Labs
MAY 6, 2025
Artificial intelligence services have been a hot topic for the last decade. It is hard to find an area or industry nowadays that hasnt at least tried to use this relatively new tool in its work. However, there is one thing that makes it possible for AI to exist. This thing is DATA. Without high-quality. AI data collection guide InData Labs.
Striim
MAY 9, 2025
Get More Insights In Your Inbox Sol Rashidi has built AI, data, and digital strategies inside some of the worlds biggest companiesand shes seen the same mistakes play out again and again. In this episode, she unpacks why AI initiatives often stall, how executives misread what transformation really requires, and why the future of AI success isnt technicalits cultural.
Snowflake
MAY 5, 2025
Consumption-based pricing helps Snowflake balance the power of true cloud elasticity with clear visibility into usage and spending. According to The State of Usage-Based Pricing: Second Edition , three out of five SaaS companies now incorporate some form of usage-based pricing into their offerings. Thats why Snowflake Ventures is reinvesting in our partner RightRev , a cloud-based revenue recognition management platform created to facilitate improved financial reporting processes, specifically
databricks
MAY 5, 2025
The imperative for modernization Traditional database solutions like SQL Server have struggled to keep up with the demands of modern data workloads due to a
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Data Engineering Weekly
MAY 11, 2025
Dagster Running Dagster: Our Open Platform We’re pulling back the curtain. Join us on May 13 for a live deep dive into how Dagster Labs runs Dagster in production. One of our lead data engineers will walk through our real-world implementation, architecture decisions, and the lessons we've learned scaling the platform. Register now Editor’s Note: OpenXData Conference - 2025 - A Free Virtual Event A free virtual event on open data architectures - Iceberg, Hudi, lakehouses, query engine
Lyft Engineering
MAY 5, 2025
Written by Josh Xi & Rakesh Kumar atLyft. From real-time rider pricing and driver incentives to long-term budget allocation and strategic planning, forecasting at Lyft plays a pivotal role in providing a foresight of our market conditions for efficient operations and facilitating millions of rides daily across North America. This article explores real-time spatial temporal forecasting models and system designs used for predicting market conditions, focusing on how their complexity and rapid
Striim
MAY 9, 2025
Introduction This recipe shows how you can build a data pipeline to read data from ServiceNow and write to BigQuery. Striim’s ServiceNow Reader will first read the existing tables from the configured ServiceNow dataset and then write them to the target BigQuery project using the BigQuery Writer, a process called “initial load” in Striim and “historical sync” or “initial snapshot” by others.
KDnuggets
MAY 12, 2025
Looking to get the most out of NotebookLM? These five expert tips will help you use it better and improve your productivity.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
databricks
MAY 5, 2025
We are experiencing an unprecedented pace of technological innovation driven by AI and data.
Snowflake
MAY 5, 2025
The automotive industry has undergone a seismic transformation over the last several decades. In 2005, most vehicles were mechanically sophisticated but had limited digital capabilities and integrations. Today, the automobile is a computer on wheels: software-defined, cloud-connected, increasingly autonomous and continuously evolving through over-the-air (OTA) updates.
ArcGIS
MAY 12, 2025
This blog article highlights advanced capabilities the ArcGIS Pro Reshape editing tool provides to modify one or more polylines or polygons. The Reshape tool advanced capabilities include previewing and choosing reshape results, and more.
Precisely
MAY 12, 2025
Better decision-making, innovation, and compliance all hinge on one common factor: trusted data. And today, were working with more data than ever. But heres a fundamental challenge that many organizations face and one Ive encountered in countless conversations with customers: they dont fully understand what data they have, let alone whether they can trust it.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
KDnuggets
MAY 12, 2025
Master LLMs through books, courses, tutorials, exercises, projects, and comprehensive guides that cover everything from foundational concepts to advanced techniques.
databricks
MAY 5, 2025
Before making architectural decisions, its worth revisiting the broader migration strategy.
Snowflake
MAY 12, 2025
We are thrilled to announce that as part of our strategic partnership with Anthropic, Snowflake customers will soon have access to Claude 3.7 Sonnet in Snowflake Cortex AI. Anthropic and Snowflake entered a multi-year partnership in November 2024 to help enterprises develop and scale easy, efficient and trusted AI products with Claude 3.5. We are building on this momentum with Claude 3.7 Sonnet.
Scott Logic
MAY 12, 2025
In this episode, Im joined by Scott Logics Sophia Ashley and Alix Dunn from the Computer Says Maybe podcast for a wide-ranging discussion which poses ethical questions that are frequently ignored in the race for technological innovation. Together, we explore the intersection of technology and power, and examine whether society is unwittingly allowing technology to reshape our world without adequate scrutiny or safeguards.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
130 
Let's personalize your content