Simon Späti

JULY 28, 2024

A Vim-Inspired Approach to Efficient Note Management with Obsidian and Markdown

Management 361

Management 361

KDnuggets

JULY 29, 2024

Learn to build the end-to-end data science pipelines from data ingestion to data visualization using Pandas pipe method.

Data Science 151

Data Science Building Data Ingestion Data 151

databricks

AUGUST 1, 2024

Today, we are excited to announce that Lakehouse Federation in Unity Catalog is now Generally Available (GA) across AWS, Azure, and GCP! Lakehouse.

AWS 139

AWS 139

Confluent

JULY 29, 2024

Apache Kafka 3.8 adds 17 new KIPs (13 for Core, 3 for Streams & 1 for Connect). Highlights include 2 new Docker images, the ability to set task assignors, and more!

Kafka 136

Kafka 136

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Seattle Data Guy

AUGUST 1, 2024

What would you do if you became the head or director of data for a 1,000-person company? Yesterday, you were plugging along as an analyst, and now, suddenly, you have all these new responsibilities. Figuring out where to start is part of the job. You’d probably feel a strong temptation to freak out. Who wouldn’t?… Read more The post How To Run A Data Team As A New Head Of Data appeared first on Seattle Data Guy.

Data 130

Data Consulting Big Data Data Analytics 130

KDnuggets

JULY 31, 2024

If you work in data, you’ll write SQL queries all the time. So how do you write efficient SQL queries that are optimized for performance? This tutorial will help you with just that.

SQL 140

SQL Data 140

Waitingforcode

AUGUST 1, 2024

Welcome to the second blog post dedicated to the previous Data+AI Summit. This time I'm going to share with you a summary of Apache Spark talks.

Data 130

Data 130

ArcGIS

JULY 31, 2024

Catch eyes and imaginations with this fun technique that draws attention to your area of interest with a bit of style!

127

127

KDnuggets

JULY 30, 2024

Follow this 7 step recipe to mastering effective insight and information dissemination through compelling data story crafting.

Data 138

Data Data Science 138

databricks

JULY 31, 2024

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient.

SQL 135

SQL Data Data Engineering Data Engineer 135

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Snowflake

AUGUST 1, 2024

Since introducing Snowpark Container Services, we’ve seen overwhelming adoption across industries from customers and partners, including Landing.AI , Relational.AI , H20.AI , SailPoint , AIR MILES , Spark NZ , and Eutelsat OneWeb. These organizations and many more are using Snowpark Container Services capabilities to easily and securely deploy everything from custom front-ends and large-scale ML training and inference to open source and homegrown models, all securely within Snowflake.

Deep Learning 123

Deep Learning Government AWS Architecture 123

Confluent

JULY 30, 2024

Confluent Platform 7.

121

121

KDnuggets

JULY 29, 2024

Let's learn how to perform memory-efficient operations in pandas with large dataset.

Datasets 137

Datasets Python 137

databricks

AUGUST 1, 2024

At Data and AI Summit, we announced the general availability of Databricks Lakehouse Monitoring. Our unified approach to monitoring data and AI.

Data 128

Data 128

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

ArcGIS

AUGUST 2, 2024

Essential Data Models in the Utility Network Foundations

Utilities 121

Utilities Data Data Management Management 121

Snowflake

AUGUST 1, 2024

Retrieval Augmented Generation (RAG) allows enterprises to ground responses from Large Language Models in their specific organization’s data. This helps ensure that AI-powered applications provide responses that are not only accurate, relevant, and consistent, but also aligned with business needs. At Snowflake, we make it simple for our customers to implement RAG, while also enabling the strict governance and privacy controls that businesses require.

Cloud 115

Cloud IT Technology Government 115

KDnuggets

AUGUST 2, 2024

This tutorial will teach you how to simplifying your file management tasks, from organization to backup, using Python’s pathlib module.

Management 134

Management Python 134

databricks

JULY 30, 2024

Introduction An organization adopting new technologies or on a modernization journey typically focuses on upcoming tools, their features and potential performance/cost improvements under.

Engineering 118

Engineering Technology Data 118

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Data Engineering Weekly

JULY 28, 2024

Meta: Introducing Llama 3.1: Our most capable models to date Probability one of the hottest announcements this week is Llama 3.1 release - the first-ever open-sourced frontier AI model competitive with leading foundation models across a range of tasks, including GPT-4, GPT-4o, and Claude 3.5 Sonnet. The Llama3 herd of models is an insightful paper that helps one deeply understand the foundational model.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

ArcGIS

JULY 29, 2024

More small-scale projected basemaps to add to the set I built in 2023

Project 104

Project 104

KDnuggets

AUGUST 1, 2024

Unlock your potential with these crafted 6 ChatGPT prompts designed to boost your productivity and streamline your operation workflows.

Designing 134

Designing 134

databricks

JULY 28, 2024

The transformative potential of artificial intelligence (AI) is undeniable. From productivity efficiency, to cost savings, and improved decision-making across all industries, AI is.

Data 116

Data 116

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Confessions of a Data Guy

AUGUST 1, 2024

The post Daft: Distributed Dataframes with Python. appeared first on Confessions of a Data Guy.

Python 100

Python Data Big Data Data Engineering 100

Snowflake

JULY 31, 2024

Academic medical centers (AMCs) are a critical keystone of healthcare systems worldwide. They serve as major hubs of medical research, pioneering new treatments that advance and set the standard of care throughout medicine. They also educate and train the next generation of healthcare professionals, ensuring that the medical field continues to advance.

Medical 98

Medical Healthcare Insurance Hospitality 98

KDnuggets

AUGUST 1, 2024

Let's learn how to use multiindex pandas for hierarchical data operations.

Data 119

Data Python 119

databricks

JULY 29, 2024

Financial Valuations & Comparative Analysis Financial institutions specialized in capital markets such as hedge funds, market makers and pension funds have long been.

103

103

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Confessions of a Data Guy

AUGUST 1, 2024

The post CI/CD for Data Engineers. appeared first on Confessions of a Data Guy.

Data Engineering 100

Data Engineering Data Engineer Engineering Data 100

Monte Carlo

JULY 30, 2024

It’s clear that data quality is becoming more of a focus for more data teams. So why are there still so many questions like these: A quick search on subreddits for data engineers, data analysts, data scientists, and more can yield a plethora of users seeking data quality advice. And while the comment below may seem like the accepted way of doing data quality management… … there’s actually a much better way.

Data Validation 96

Data Validation Datasets Medical Raw Data 96

KDnuggets

JULY 30, 2024

Learning how to perform several of the most basic matrix operations with NumPy.

Python 114

Python 114

databricks

JULY 31, 2024

Companies across all industries want to share data with each other to enable collaboration and accelerate innovation. However, these organizations often use different.

Data 94

Data 94

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer