KDnuggets

AUGUST 6, 2024

Learn statistics through interactive books, code examples, cheat sheets, guides, and tools documentation.

Coding 145

Coding Data Science Data 145

databricks

AUGUST 6, 2024

Fueled by the exponential growth in external data and AI for innovation, organizations across all industries are looking for effective ways to collaborate.

Data 140

Data 140

ArcGIS

AUGUST 6, 2024

Explore how moving from ArcMap to ArcGIS Pro and user types can make GIS workflows better, improve collaboration, and make big changes within your organization.

128

128

Data Engineering Weekly

AUGUST 6, 2024

TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

AUGUST 6, 2024

Harness the simplicity and effectiveness of Hugging Face's Datasets library to efficiently load datasets, regardless of their source

Datasets 86

Datasets Data 86

Cloudera

AUGUST 6, 2024

Over the past several years, data leaders asked many questions about where they should keep their data and what architecture they should implement to serve an incredible breadth of analytic use cases. Vendors with proprietary formats and query engines made their pitches, and over the years the market listened, and data leaders made their decisions. The most interesting thing about their choices is that, despite the millions of marketing dollars vendors spent trying to convince customers that the

Metadata 82

Metadata Government Datasets Architecture 82

Confluent

AUGUST 6, 2024

Using SQL-based BigQuery Continuous Queries w/Confluent lets you stream your warehouse data in real-time, sending it downstream for analytics use cases & more.

SQL 69

SQL Data IT Cloud 69

Yelp Engineering

AUGUST 6, 2024

We’re excited to announce that multi-metric horizontal autoscaling is available for all services at Yelp. This allows us to scale services using multiple metrics, such as the number of in-flight requests and CPU utilization, rather than relying on a single metric. We expect this to provide us with better resilience and faster recovery during outages.

Coding 52

Coding Utilities Management 52

Monte Carlo

AUGUST 6, 2024

Data warehouses are the centralized repositories that store and manage data from various sources. They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. However, beneath their surface lies a host of invisible risks embedded within the data warehouse layers. These “hidden threats” can silently undermine your data quality and reliability and often remain undetected until they trigger significant problems such as incorrect busines

Data Warehouse 52

Data Warehouse Raw Data Machine Learning BI 52

DataKitchen

AUGUST 6, 2024

Chris Bergh joins me to chat about all things DataOps. We also discuss lean, removing waste from data processes and teams, and much more.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

RandomTrees

AUGUST 6, 2024

Manufacturing has always been at the cutting edge of technology since it drives economic growth and societal changes. As a result, in recent times, the development of Generative Artificial Intelligence (GenAI) has opened up new possibilities for innovation in this critical area. GenAI is an artificial intelligence subset dedicated to generating new content and designs.

Manufacturing 52

Manufacturing Transportation Datasets High Quality Data 52

DataKitchen

AUGUST 6, 2024

Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to…

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Hevo

AUGUST 6, 2024

In today’s dynamic business environment, companies often need to migrate their databases for many different reasons, ranging from scaling their operations to modernizing their technology stack or moving to the cloud to enjoy numerous benefits.

AWS 52

AWS Database Cloud Technology 52

DataKitchen

AUGUST 6, 2024

DataKitchen’s Data Quality TestGen found 18 potential data quality issues in a few minutes (including install time) on data.boston.gov building permit data! Imagine a free tool that you can point at any dataset and find actionable data quality issues immediately! It sure beats having your data consumers tell you about problems they find when you are trying to enjoy your weekend.

Datasets 50

Datasets SQL Data Building 50

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Hevo

AUGUST 6, 2024

Introduction In the data-driven modern world, organizations are quite dependent on ETL tools that help them integrate their data efficiently. These are the tools that base their guarantee of a smooth flow of data from sources to destination for supporting businesses in making decisions.

ETL Tools 52

ETL Tools Data 52

Hevo

AUGUST 6, 2024

We’ve all been there: your business is growing, and your data is expanding across various systems. You’re trying to keep everything in sync, but manual updates and batch processing don’t cut it anymore. You need a reliable way to keep your data up-to-date across all platforms.

AWS 52

AWS SQL Database Systems 52