KDnuggets

FEBRUARY 18, 2025

Develop AI applications, test them, and deploy on the cloud using user-friendly MLOps tools and straightforward methods.

Project 128

Project Cloud 128

Waitingforcode

FEBRUARY 18, 2025

Using cloud managed services is often a love and hate story. On one hand, they abstract a lot of tedious administrative work to let you focus on the essentials. From another, they often have quotas and limits that you, as a data engineer, have to take into account in your daily work. These limits become even more serious when they operate in a latency-sensitive context, as the one of stream processing.

Data Engineering 130

Data Engineering Data Engineer Cloud Data 130

Data Engineering Weekly

FEBRUARY 18, 2025

Fluss is a compelling new project in the realm of real-time data processing. I spoke with Jark Wu , who leads the Fluss and Flink SQL team at Alibaba Cloud, to understand its origins and potential. Jark is a key figure in the Apache Flink community, known for his work in building Flink SQL from the ground up and creating Flink CDC and Fluss. You can read the Q&A version of the conversation here, and don’t forget to listen to the podcast.

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Snowflake

FEBRUARY 18, 2025

Being able to leverage unstructured data is a critical part of an effective data strategy for 2025 and beyond. To keep up with the competition and AI-accelerated pace of innovation, businesses must be able to mine the treasure trove of value buried in the mountains of unstructured data that comprise approximately 80% of all enterprise data from call center logs, customer reviews, emails and claims reports to news, filings and transcripts.

Unstructured Data 72

Unstructured Data Insurance Structured Data Government 72

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

databricks

FEBRUARY 18, 2025

As we welcome the new year, we're thrilled to announce several new resources for R users on Databricks: a comprehensive developer guide, the.

Engineering 107

Engineering 107

KDnuggets

FEBRUARY 18, 2025

In this article we will go through the tips and tricks that can help with your logic-building skills.

Programming 95

Programming Building 95

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis. However, these two processes are essentially distinct, and their testing needs differ in manyways.

Data Pipeline 52

Data Pipeline SQL Raw Data Python 52

Snowflake

FEBRUARY 18, 2025

2025AIE 80% AI 3 AI ESGLLM AIAI Snowflake AISnowflakeAISnowflake Cortex AI Cortex AIAIAICortex AILLMRAG GPUCortex AIGoogleAnthropicMetaMistral AIAI1 SnowflakeAIROI AI Blueprint for Financial Services Accelerate

52

52

databricks

FEBRUARY 18, 2025

Introduction VisitBritain is the official website for tourism to the United Kingdom, designed to help visitors plan their trips and get recommendations on.

Designing 52

Designing 52

Engineering at Meta

FEBRUARY 18, 2025

Metas Anti Scraping team focuses on preventing unauthorized scraping as part of our ongoing work to combat data misuse. In order to protect Metas changing codebase from scraping attacks, we have introduced static analysis tools into our workflow. These tools allow us to detect potential scraping vectors at scale across our Facebook, Instagram, and even parts of our Reality Labs codebases.

Coding 117

Coding Data Python Systems 117

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Yelp Engineering

FEBRUARY 18, 2025

Background As Yelps business continues to grow, the revenue streams have become more complex due to the increased number of transactions, new products and services. These changes over time have challenged the manual processes involved in Revenue Recognition. As described in the first post of the Revenue Automation Series, Yelp invested significant resources in modernizing its Billing System to fulfill the pre-requisite of automating the revenue recognition process.

Data Pipeline 87

Data Pipeline Building Data Systems 87

dbt Developer Hub

FEBRUARY 18, 2025

Remember how dbt felt when you had a small project? You pressed enter and stuff just happened immediately? We're bringing that back. Benchmarking tip: always try to get data that's good enough that you don't need to do statistics on it After a series of deep dives into the guts of SQL comprehension , let's talk about speed a little bit. Specifically, I want to talk about one of the most annoying slowdowns as your project grows: project parsing.

Engineering 71

Engineering Python SQL Project 71

Striim

FEBRUARY 18, 2025

As organizations increasingly rely on AI to drive innovation and efficiency, protecting sensitive data has become both a strategic necessity and a regulatory mandate. Traditional security measures, often reactive and manual, no longer suffice. Instead, we now stand at the cusp of a new era where data governance is automatic, intelligent, and built to match the speed of AI.

Data Governance 52

Data Governance Government NoSQL Unstructured Data 52

Sync Computing

FEBRUARY 18, 2025

For data teams, understanding the true cost of operations has always been a complex puzzle. This is because your monthly bills come from multiple sources. For example, when using Databricks you have: The Databricks bill for DBU consumption Your cloud provider’s bill (AWS, Azure, or GCP) for the infrastructure powering your Databricks workloads This fragmented view makes it challenging to understand your true total cost of ownership (TCO).

Cloud 52

Cloud AWS Cloud Computing Management 52

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems