KDnuggets

JULY 12, 2024

Discover the essential tools every data scientist should know to elevate their data science game, from Python and R to SQL and advanced visualization tools.

Data Science 150

Data Science SQL Python Data 150

databricks

JULY 8, 2024

We are proud to announce two new analyst reports recognizing Databricks in the data engineering and data streaming space: IDC MarketScape: Worldwide Analytic.

Data Pipeline 137

Data Pipeline Process Cloud Data Engineering 137

Snowflake

JULY 9, 2024

Snowflake is committed to helping customers protect their accounts and data. That’s why we have been working on product capabilities that allow Snowflake admins to make multifactor authentication (MFA) mandatory and monitor compliance with this new policy. As part of that effort, today we’re announcing several key features: 1. A new authentication policy that requires MFA for all users in a Snowflake account 2.

Technology 135

Technology Accessible Accessibility Management 135

Waitingforcode

JULY 10, 2024

Welcome to the first Data+AI Summit 2024 retrospective blog post. I'm opening the series with the topic close to my heart at the moment, stream processing!

Data 130

Data Process 130

Advertisement

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

Datasets

KDnuggets

JULY 9, 2024

This article is for anyone looking to maximize their use of Amazon Web Services (AWS) generative AI (GenAI) services. Here are eight courses that range from beginner to expert level.

AWS 140

AWS Amazon Web Services 140

ArcGIS

JULY 12, 2024

Creation of a Digital Twin in Seven Days with ArcGIS in Zurich

Government 122

Government 122

Confessions of a Data Guy

JULY 9, 2024

When I was young and full of myself, writing Perl and PHP, while your ma was still reading you a bedtime story and giving you a stuffy to fall asleep with, I had to program uphill, both ways, in the rain and snow. Not like you milk toast Data Engineers clickty clicking around Databricks and […] The post The Abstractions Are Making You Dumb (rise of the Shallow Expert) appeared first on Confessions of a Data Guy.

Data Engineering 114

Data Engineering Data Engineer Programming Engineering 114

KDnuggets

JULY 8, 2024

Text preprocessing is an important step in NLP. Let's learn how to use the Hugging Face Tokenizers Library to preprocess text data.

Data 129

Data 129

databricks

JULY 12, 2024

Hallucinations in large language models (LLMs) occur when models produce responses that do not align with factual reality or the provided context. This.

126

126

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

ArcGIS

JULY 12, 2024

Esri is working with partners (Maxar, TomTom) to enhance our 3D basemaps with high-quality commercial data for elevation and buildings layers.

Building 117

Building Data 117

Engineering at Meta

JULY 10, 2024

Meta’s advertising business leverages large-scale machine learning (ML) recommendation models that power millions of ads recommendations per second across Meta’s family of apps. Maintaining reliability of these ML systems helps ensure the highest level of service and uninterrupted benefit delivery to our users and advertisers. To minimize disruptions and ensure our ML systems are intrinsically resilient, we have built a comprehensive set of prediction robustness solutions that ensure stability w

Machine Learning 112

Machine Learning Algorithm Systems Engineering 112

KDnuggets

JULY 10, 2024

Let's learn how to efficiently merge large Pandas dataframes.

Python 121

Python 121

Precisely

JULY 12, 2024

Data can be your organization’s most valuable asset, but only if it’s data you can trust. When companies work with data that is untrustworthy for any reason, it can result in incorrect insights, skewed analysis, and reckless recommendations to become data integrity vs data quality. Two terms can be used to describe the condition of data: data integrity and data quality.

Data Integration 105

Data Integration Datasets Data Data Governance 105

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

ArcGIS

JULY 12, 2024

You can host scene layers and 3D tiles layers in ArcGIS Online or reference datasets in cloud storage in ArcGIS Enterprise.

Cloud Storage 103

Cloud Storage Datasets Cloud Data Management 103

Cloudera

JULY 10, 2024

There’s nothing worse than wasting money on unnecessary costs. In on-premises data estates, these costs appear as wasted person-hours waiting for inefficient analytics to complete, or troubleshooting jobs that have failed to execute as expected, or at all. They manifest as idle hardware waiting for urgent workloads to come in, ensuring sufficient spare capacity to run them amidst noisy neighbors and resource-hungry, lower-priority workloads.

Metadata 99

Metadata Cloud Management IT 99

KDnuggets

JULY 11, 2024

Learn data science through interactive courses, books, guides, code examples, projects, and free courses based on top university curricula. Also, access interview questions and best practices.

Data Science 116

Data Science Data Coding Accessible 116

Engineering at Meta

JULY 10, 2024

Tail utilization is a significant system issue and a major factor in overload-related failures and low compute utilization. The tail utilization optimizations at Meta have had a profound impact on model serving capacity footprint and reliability. Failure rates, which are mostly timeout errors, were reduced by two-thirds; the compute footprint delivered 35% more work for the same amount of resources; and p99 latency was cut in half.

Utilities 98

Utilities Machine Learning Architecture Systems 98

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Notion: Building and scaling Notion’s data lake Notion writes about scaling the data lake by bringing critical data ingestion operations in-house.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Snowflake

JULY 9, 2024

There is a scene in Mission: Impossible – Rogue Nation where Tom Cruise is hanging onto the outside of a jet as it has taken off. And while, yes, he’s going with it, he’s not really on board or in control. Some data executives feel like that. It’s not enough to establish goals — or, the destination in this metaphor. The data strategy must provide a flight plan for making sure you get there — on time, on budget and, of course, safely on board.

Government 98

Government Education Data Governance Unstructured Data 98

KDnuggets

JULY 8, 2024

Docker tags are important for managing and versioning Docker images. This tutorial will teach you how to use Docker tags effectively.

Management 113

Management Data Engineering Data Engineer Engineering 113

Ripple Engineering

JULY 9, 2024

Introduction: Embracing the Future with Ripple's Data Platform Migration Welcome to a pivotal moment in Ripple's data journey. As leaders at the intersection of blockchain technology and financial services, we're excited to share a transformative step in our data management evolution. We recently embarked on a significant data platform migration, transitioning from Hadoop to Databricks, a move motivated by our relentless pursuit of excellence and our contributions to the XRP Ledge

Hadoop 96

Hadoop Data Lake Machine Learning Raw Data 96

Advertisement

With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines. Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. This introductory tutorial provides a crash course for writing and deploying your first Airflow pipeline.

Cloud

Uber Engineering

JULY 11, 2024

Modernizing the fundamentals of log management at Uber: How we used CLP to build a new logging infra that lets users view and analyze their logs seamlessly, at scale!

Building 81

Building Management Engineering 81

Snowflake

JULY 11, 2024

Leaders in the advertising, media and entertainment industries know all too well the importance of the media supply chain. It’s the backbone that keeps things running smoothly, including everything from content creation and management to content distribution and analytics. But media supply chains are becoming more complex to manage for several reasons.

Media 83

Media Entertainment Cloud Metadata 83

KDnuggets

JULY 11, 2024

Here’s how to ace your data analyst interview and land your first job.

Data 110

Data 110

Confluent

JULY 11, 2024

confluent-kafka-javascript offers two APIs for calling Kafka from JS, one KafkaJS-like & one node-rdkafka-like. Learn it by making an app w/producer, consumer & UI.

Kafka 78

Kafka Building IT 78

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Uber Engineering

JULY 6, 2024

Kafka Tiered Storage, developed in collaboration with the Apache Kafka community, introduces the separation of storage and processing in brokers, significantly improving the scalability, reliability, and efficiency of Kafka clusters.

Kafka 75

Kafka Process Data 75

Towards Data Science

JULY 12, 2024

Data as a product is an intriguing concept, but beware of the application trap Continue reading on Towards Data Science »

Data Science 75

Data Science Data Data Engineering Data Engineer 75

KDnuggets

JULY 8, 2024

Learn all about introductory statistics with this collection of tutorials from our sister site Statology.

Data Science 109

Data Science Data 109