KDnuggets

JANUARY 6, 2025

AI is growing fast! Learn the top skills for 2025 to stay ahead in this exciting field.

130

130

KDnuggets

JANUARY 6, 2025

Kickstart your data engineering career with an expert guide available on GitHub.

Data Engineering 124

Data Engineering Data Engineer Engineering Data 124

Hevo

JANUARY 9, 2025

Have you ever felt like data engineering is evolving at the speed of light? With new tech emerging almost daily, it’s no surprise that staying ahead of the curve is harder than ever. As we step into the fantastic year 2025 ahead, the rate at which data engineering changes is at an all-time high.

Data Engineering 40

Data Engineering Data Engineer Engineering Data 40

Snowflake

JANUARY 8, 2025

Though AI is (still) the hottest technology topic, its not the overriding issue for enterprise security in 2025. Advanced AI will open up new attack vectors and also deliver new tools for protecting an organizations data. But the underlying challenge is the sheer quantity of data that overworked cybersecurity teams face as they try to answer basic questions such as, Are we under attack?

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Netflix Tech

JANUARY 6, 2025

This article is the last in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. Need to catch up? Check out Part 1 , which detailed how were empowering Netflix to efficiently produce and effectively deliver high quality, actionable analytic insights across the company and Part 2 , which stepped through a few exciting business applications for Analytics Engineering.

Engineering 81

Engineering Entertainment Designing Finance 81

KDnuggets

JANUARY 8, 2025

Element-wise operations are a crucial part of data preprocessing in Pandas. Learn how to perform them with practical examples using the DataFrame.map() function.

Data 122

Data 122

Data Engineering Weekly

JANUARY 8, 2025

What if your data lake could do more than just store information—what if it could think like a database? As data lakehouses evolve, they transform how enterprises manage, store, and analyze their data. To explore this future, I recently sat down with Vinoth Chandar, founder of Onehouse and creator of Apache Hudi, for a fireside chat about the trends shaping the data landscape.

Data Lake 62

Data Lake Datasets Retail Data Ingestion 62

ArcGIS

JANUARY 7, 2025

Must read article that introduces the SAR analysis toolset in ArcGIS Pro, which helps users extract valuable insights from processed SAR data.

Process 104

Process Education Data 104

KDnuggets

JANUARY 7, 2025

Do you want to learn data wrangling with Python on a budget? No worries, there are (at least) five free courses thatll provide you with solid knowledge.

Python 121

Python Data 121

Precisely

JANUARY 9, 2025

Artificial Intelligence (AI) is all the rage, and rightly so. By now most of us have experienced how Gen AI and the LLMs (large language models) that fuel it are primed to transform the way we create, research, collaborate, engage, and much more. Yet along with the AI hype and excitement comes very appropriate sanity-checks asking whether AI is ready for prime-time.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Confessions of a Data Guy

JANUARY 10, 2025

Every once in a great while, the question comes up: “How do I test my Databricks codebase?” It’s a fair question, and if you’re new to testing your code, it can seem a little overwhelming on the surface. However, I assure you the opposite is the case. Testing your Databricks codebase is no different than […] The post Testing and Development for Databricks Environment and Code. appeared first on Confessions of a Data Guy.

Coding 113

Coding IT Data 113

Netflix Tech

JANUARY 6, 2025

Part 2: Navigating Ambiguity By: VarunKhaitan With special thanks to my stunning colleagues: Mallika Rao , Esmir Mesic , HugoMarques Building on the foundation laid in Part 1 , where we explored the what behind the challenges of title launch observability at Netflix, this post shifts focus to the how. How do we ensure every title launches seamlessly and remains discoverable by the right audience?

Metadata 88

Metadata Algorithm Systems Building 88

Cloudera

JANUARY 7, 2025

Airports are an interconnected system where one unforeseen event can tip the scale into chaos. For a smaller airport in Canada, data has grown to be its North Star in an industry full of surprises. In order for data to bring true value to operationsand ultimately customer experiencesthose data insights must be grounded in trust. Ryan Garnett, Senior Manager Business Solutions of Halifax International Airport Authority, joined The AI Forecast to share how the airport revamped its approach to data

Finance 81

Finance Building Cloud Engineering 81

KDnuggets

JANUARY 9, 2025

Curious about what LLMs are and want to know about them? Explore the Full Guide Right Here, Right Now!

147

147

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Confessions of a Data Guy

JANUARY 7, 2025

Building fun things is a real part of Data Engineering. Using your creative side when building a Lake House is possible, and using tools that are outside the normal box can sometimes be preferable. Checkout this video where I dive into how I build just such a Lake House using Modern Data Stack tools like […] The post Building a Fast, Light, and CHEAP Lake House with DuckDB, Delta Lake, and AWS Lambda appeared first on Confessions of a Data Guy.

AWS 130

AWS Building Data Engineering Data Engineer 130

Waitingforcode

JANUARY 9, 2025

Time travel is a quite popular Delta Lake feature. But do you know it's not the single one you can use to interact with the past versions? An alternative is the RESTORE command, and it'll be the topic of this blog post.

IT 130

IT 130

Cloudera

JANUARY 9, 2025

A Name That Matches the Moment For years, Clouderas platform has helped the worlds most innovative organizations turn data into action. As the AI landscape evolves from experiments into strategic, enterprise-wide initiatives, its clear that our naming should reflect that shift. Thats why were moving from Cloudera Machine Learning to Cloudera AI. This isnt just a new label or even AI washing.

Machine Learning 85

Machine Learning Raw Data Government Algorithm 85

KDnuggets

JANUARY 6, 2025

Learn how to structure your data science projects to make them more organized and minimize chaos!

Data Science 128

Data Science Project Data 128

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Confessions of a Data Guy

JANUARY 8, 2025

You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. It could be Data Quality, Data Modeling, Data Pipelines, Data Storage, Compute, and […] The post What is a Data Platform?

Data Storage 124

Data Storage Data Pipeline Data IT 124

Waitingforcode

JANUARY 5, 2025

Even though I was blogging less in the second half of the previous year, the retrospective is still the blog post I'm waiting for each year. Every year I summarize what happened in the past 12 months and share with you my future plans. It's time for the 2024 Edition!

IT 130

IT 130

databricks

JANUARY 7, 2025

At Databricks, our automation vision is to automate all aspects of the business, making it better, faster, and cheaper. For the sales teams.

IT 115

IT Machine Learning 115

KDnuggets

JANUARY 10, 2025

This guide highlights the importance of container monitoring, key metrics to track, and tools ranging from Docker's built-in commands to comprehensive systems like Prometheus and Grafana.

Systems 125

Systems 125

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

ArcGIS

JANUARY 10, 2025

Address spatial confounding with Create Spatial Component Explanatory Variables in ArcGIS Pro 3.

Machine Learning 108

Machine Learning 108

Monte Carlo

JANUARY 6, 2025

2024 was a real doozy. If you emerged from the generative AI haze with your sanity still intact, then we salute you. This year, we saw early GenAI use cases like chatbots and copilots, we saw data teams introducing open table formats into their lakehouses, we saw data products grow in popularity more than ever before, and we saw everything in between.

Insurance 52

Insurance Unstructured Data High Quality Data Government 52

databricks

JANUARY 6, 2025

Introduction MLOps is an ongoing journey, not a once-and-done project. It involves a set of practices and organizational behaviors, not just individual tools.

Project 114

Project IT 114

KDnuggets

JANUARY 9, 2025

Five years ago, I was a data science beginner learning the ropes. If I could start anew in 2025, here's what I would do.

Data Science 122

Data Science Data 122

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Hevo

JANUARY 5, 2025

The trend of today’s information-driven world is to make decisions based on information. The human resources departments are not left behind in this trend. Integration of HR data has become an important step in smoothing the flow of HR processes, improving the employee experience, and ensuring compliance in a technology-enabled environment.

Data Integration 40

Data Integration Data Technology Process 40

Towards Data Science

JANUARY 7, 2025

Building idempotent and re-playable data pipelines Continue reading on Towards Data Science

Data Pipeline 84

Data Pipeline Data Science Data Building 84

databricks

JANUARY 7, 2025

We are excited to announce that egress control for Databricks serverless and Mosaic AI Model Serving workloads is available in Public Preview on.

111

111

KDnuggets

JANUARY 10, 2025

Build your own state-of-the-art AI applications with simple steps and beginner-friendly methodologies.

Project 121

Project Building 121

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer