Knowledge Hut

NOVEMBER 2, 2023

Artificial Intelligence has transformed the way we tackle intricate problems, interpret data, and make forecasts, revolutionizing the tech realm with its uninhabited prowess and potential. In fact, did you know that the global market for AI , which currently stands at a market value of $150.2 billion, is expected to witness a 36.8% CAGR by the end of 2030?

Machine Learning 52

Machine Learning Deep Learning Utilities Medical 52

Knowledge Hut

NOVEMBER 2, 2023

For those aspiring to build a career within the Azure ecosystem, navigating the choices between Azure Data Engineers and Azure DevOps Engineers can be quite challenging. Azure Data Engineers and Azure DevOps Engineers are two critical components of the Azure ecosystem for different but interconnected reasons. A choice between these two can be difficult to make unless you have all the information you need.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

The Pragmatic Engineer

OCTOBER 31, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover three out of seven topics from today’s subscriber-only issue Three Cloud Providers, Three Outages: Three Different Responses.

AWS 290

AWS Google Cloud Cloud Engineering 290

Data Engineering Podcast

OCTOBER 29, 2023

Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection.

Database 189

Database SQL BI Machine Learning 189

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

NOVEMBER 1, 2023

This list of machine learning algorithms is a good place to start your journey as a data scientist. You should be able to identify the most common models and use them in the right applications.

Machine Learning 158

Machine Learning Algorithm Data 158

databricks

OCTOBER 31, 2023

Today we're excited to announce MLflow 2.8 supports our LLM-as-a-judge metrics which can help save time and costs while providing an approximation of.

Data Science 145

Data Science Engineering Data 145

Snowflake

NOVEMBER 2, 2023

Snowflake’s single, cross-cloud governance model has always been a powerful differentiator, enabling customers to manage their increasingly complex data ecosystems with simplicity and ease. As a result, Snowflake is enhancing its governance capabilities that thousands of customers already rely on through Snowflake Horizon. Snowflake Horizon is Snowflake’s built-in governance solution with a unified set of compliance, security, privacy, interoperability, and access capabilities.

Metadata 129

Metadata Government AWS Medical 129

KDnuggets

NOVEMBER 2, 2023

Unlock the Power of SQL in Data Visualization: Master the Art of Preparing Data for Impactful Charts and Graphs.

SQL 150

SQL Data 150

databricks

OCTOBER 30, 2023

Introduction Four months ago, we shared how AMD had emerged as a capable platform for generative AI and demonstrated how to easily and.

Data Science 144

Data Science Engineering Data 144

Confluent

NOVEMBER 3, 2023

Learn how AI-powered customer intelligence platform Dialpad modernized its data infrastructure and improved customer satisfaction rates with Confluent and Startree.

IT 127

IT Data 127

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Snowflake

NOVEMBER 1, 2023

Generative AI (GenAI) and large language models (LLMs) are disrupting the way we work at a global scale. Snowflake is excited to announce an innovative product lineup that brings our platform’s ease of use, security and governance to the GenAI world. Through these new offerings, any user can incorporate LLMs into analytical processes in seconds; developers can create GenAI-powered apps in minutes, or within hours execute powerful workflows, like fine-tuning foundation models on enterprise data —

Government 127

Government SQL AWS Database 127

KDnuggets

NOVEMBER 2, 2023

For production grade LLM apps, you need a robust data pipeline. This article talks about the different stages of building a Gen AI data pipeline and what is included in these stages.

Data Pipeline 148

Data Pipeline Building Data 148

databricks

NOVEMBER 1, 2023

Whether you’re an NFL fanatic, an alumnus rooting for your alma mater or a super fan just trying to catch a glimpse of T.

Data 131

Data 131

Tweag

NOVEMBER 1, 2023

This blog post is part of the series exploring the foundations of the Nickel configuration language. Presenting Nickel: better configuration for less Programming with contracts in Nickel Types à la carte in Nickel Great configurations from little merges grow We previously looked at the core language, then contracts, and finally typing. The last important remaining piece to explore is the merge system.

Metadata 117

Metadata Programming Programming Language Coding 117

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Snowflake

NOVEMBER 1, 2023

Generative AI is unlocking new ways to drive innovation, improve productivity and derive more value from data. For organizations to fully capitalize on this potential, it’s critical that everyone — not just those with AI expertise — is able to access and use generative AI. That’s why we created Snowflake Cortex (in private preview), Snowflake’s new, intelligent, fully managed service that enables organizations to quickly analyze data and build AI applications — all within Snowflake.

Unstructured Data 126

Unstructured Data SQL Python Accessible 126

KDnuggets

NOVEMBER 3, 2023

Learn how to tune your model’s hyperparameters using grid search and randomized search. Also learn to implement them in scikit-learn using GridSearchCV and RandomizedSearchCV.

Machine Learning 146

Machine Learning 146

databricks

OCTOBER 30, 2023

Last year, we published the Big Book of MLOps, outlining guiding principles, design considerations, and reference architectures for Machine Learning Operations (MLOps). Since.

Machine Learning 122

Machine Learning Architecture Designing Data Science 122

ArcGIS

NOVEMBER 2, 2023

Sometimes mosaiced imagery over water can have undesirable seam artifacts or gaps. Here's one way to make that water look snazzy.

113

113

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Snowflake

NOVEMBER 2, 2023

In the ever-evolving world of data management, Snowflake is at the forefront of empowering our customers to make informed decisions about data while ensuring cost efficiency and control. Admins know that managing and optimizing platform costs can be a complex and time-consuming task. To help them more intuitively understand, control and optimize spend from one centralized place, Snowflake is introducing the new Cost Management Interface (private preview).

Management 124

Management Retail SQL AWS 124

KDnuggets

OCTOBER 31, 2023

The future of the multimodal large language model.

146

146

Cloudera

NOVEMBER 1, 2023

Elevate your AI applications with our latest applied ML prototype At Cloudera, we continuously strive to empower organizations to unlock the full potential of their data, catalyzing innovation and driving actionable insights. And so we are thrilled to introduce our latest applied ML prototype (AMP) — a large language model (LLM) chatbot customized with website data using Meta’s Llama2 LLM and Pinecone’s vector database.

Machine Learning 110

Machine Learning Data Ingestion Database Architecture 110

ArcGIS

OCTOBER 31, 2023

There are many imagery sources available to host your own imagery layers in ArcGIS Image for ArcGIS Online.

Data 112

Data 112

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Snowflake

NOVEMBER 1, 2023

Snowflake is announcing new product capabilities that are changing how developers build, deliver, distribute and operate their applications. These new features include programming language and hardware flexibility from Snowpark Container Services, as well as the ability to build, distribute and monetize full-stack apps with the Snowflake Native App Framework; the ability to leverage transactional and analytical data together with Hybrid Tables; and DevOps capabilities including database change m

AWS 123

AWS Database Programming Language Data Science 123

KDnuggets

NOVEMBER 3, 2023

Discover the truth behind popular AI myths and dive deep into the genuine capabilities and impact of Generative AI in today's world.

145

145

Netflix Tech

NOVEMBER 3, 2023

Democratizing Stream Processing @ Netflix By Guil Pires , Mark Cho , Mingliang Liu , Sujay Jain Data powers much of what we do at Netflix. On the Data Platform team, we build the infrastructure used across the company to process data at scale. In our last blog post, we introduced “Data Mesh” — A Data Movement and Processing Platform. When a user wants to leverage Data Mesh to move and transform data, they start by creating a new Data Mesh pipeline.

SQL 109

SQL Pipeline-centric Kafka Data 109

ArcGIS

NOVEMBER 3, 2023

ArcGIS GeoEnrichment Service quickly adds information like local demographics, spending patterns, and business data to your study area.

Data 109

Data 109

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Snowflake

NOVEMBER 1, 2023

Snowflake has invested heavily in extending the Data Cloud to AI/ML workloads, starting in 2021 with the introduction of Snowpark , the set of libraries and runtimes in Snowflake that securely deploy and process Python and other popular programming languages. Since then, we’ve significantly opened up the ways Snowflake’s platform, including its elastic compute engine can be used to accelerate the path from AI/ML development to production.

Building 119

Building Python SQL Programming Language 119

KDnuggets

NOVEMBER 1, 2023

Whether you're doing machine learning, scientific computing, or working with huge datasets, CuPy is an absolute game-changer.

Python 145

Python Datasets Machine Learning 145

databricks

NOVEMBER 2, 2023

In this blog we will demonstrate with examples, how you can seamlessly upgrade your Hive metastore (HMS)* tables to Unity Catalog (UC) using.

105

105

Cloudera

OCTOBER 31, 2023

Enterprises see embracing AI as a strategic imperative that will enable them to stay relevant in increasingly competitive markets. However, it remains difficult to quickly build these capabilities given the challenges with finding readily available talent and resources to get started rapidly on the AI journey. Cloudera recently signed a strategic collaboration agreement with Amazon Web Services (AWS), reinforcing our relationship and commitment to accelerating and scaling cloud native data manag

Building 104

Building Amazon Web Services AWS Machine Learning 104

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer