KDnuggets

AUGUST 23, 2024

Learn large language models with these free resources from Deeplearning.AI, Microsoft, AWS, and more.

AWS 148

AWS 148

ArcGIS

AUGUST 23, 2024

With a new GIS mapping tool you can map the most visited national parks (and much more!) to explore your spatial data even further.

Designing 145

Designing Data 145

databricks

AUGUST 22, 2024

Today, we're excited to announce the launch of Data Warehouse Brickbuilder Migration Solutions. This is an expansion to the Brickbuilder Program , which.

Data Warehouse 132

Data Warehouse Data Programming 132

Waitingforcode

AUGUST 22, 2024

Code organization and assertions flow are both important but even them, they can't guarantee your colleagues' adherence to the unit tests. There are other user-facing attributes to consider as well.

Coding 130

Coding 130

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

AUGUST 22, 2024

Have you ever wanted to create your library in Python? Well, it’s achievable and surprisingly quite easy!

Python 140

Python 140

ArcGIS

AUGUST 21, 2024

Does your map need a north arrow? It depends.

IT 136

IT Designing 136

Confessions of a Data Guy

AUGUST 20, 2024

Recently, for some unknown reason, I was pursuing the new Stackoverflow … called Reddit, for Data Engineering … and I ran across an interesting question … more or less it was related to “what makes a good Software Engineer … in a Data Engineering context.” This isn’t the first time this idea has come up […] The post What is a “Good” Data or Software Engineer?

Software Engineer 113

Software Engineer Software Engineering Engineering Data Engineer 113

KDnuggets

AUGUST 19, 2024

Improve your Pandas processing with the NumPy package.

Data Analysis 139

Data Analysis Data Process Python 139

ArcGIS

AUGUST 21, 2024

Discover how ArcGIS empowers users at all levels to harness the potential of geospatial AI. Whether you're leveraging pre-trained models for quick insights or building custom AI solutions, ArcGIS offers flexible, powerful tools for every workflow. Explore simplified and advanced AI capabilities across desktop, enterprise, and cloud environments, designed to make geospatial intelligence accessible to everyone.

Cloud 124

Cloud Designing Accessible Accessibility 124

databricks

AUGUST 20, 2024

Special thanks to Kevin Glover, Martin Ko, Kuber Sharma and the team at Tableau for their valuable insights and contributions to this blog.

116

116

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Engineering at Meta

AUGUST 20, 2024

Delivering new AI technologies at scale also means rethinking every layer of our infrastructure – from silicon and software systems and even our data center designs. For the second year in a row, Meta’s engineering and infrastructure teams returned for the AI Infra @ Scale conference, where they discussed the challenges of scaling up an infrastructure for AI as well as work being done on our large-scale GPU clusters , open hardware designs for next-generation data center hardware, and how Meta i

Engineering 104

Engineering Designing Technology Systems 104

KDnuggets

AUGUST 22, 2024

The article highlights five top free machine learning courses to enhance your skills.

Machine Learning 135

Machine Learning 135

ArcGIS

AUGUST 21, 2024

Did you catch the Huff model demo at UC 2024? Read this article for step-by-step instructions on the Business Analyst workflow.

Business Analyst 113

Business Analyst 113

databricks

AUGUST 19, 2024

We are excited to announce the latest addition to the Databricks developer experience: the PyCharm Professional Integration with Databricks ! This new plugin.

107

107

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Engineering at Meta

AUGUST 23, 2024

Learn about new PyTorch advancements for LLMs and how PyTorch is enhancing every aspect of the LLM lifecycle. In this talk from AI Infra @ Scale 2024 , software engineers Wanchao Liang and Evan Smothers are joined by Meta research scientist Kimish Patel to discuss our newest features and tools that enable large-scale training, memory efficient fine-tuning, and on-device LLM capabilities.

Software Engineer 103

Software Engineer Software Engineering Algorithm Architecture 103

KDnuggets

AUGUST 20, 2024

Learn how to build the recommendation system with advanced technology.

Systems 133

Systems Building Technology Machine Learning 133

Edureka

AUGUST 19, 2024

If you’re curious about learning DevOps Roadmap but don’t know where to start, you’re in the right place! I’ll guide you step-by-step on your journey to becoming a DevOps engineer. First and foremost, we will start with the basic skills required to become a DevOps engineer, then gradually explore the major milestones you need to reach to succeed in this field.

Engineering 98

Engineering Programming Language Python Cloud 98

databricks

AUGUST 22, 2024

Databricks Workflows is the cornerstone of the Databricks Data Intelligence Platform, serving as the orchestration engine that powers critical data and AI workloads.

Engineering 85

Engineering Data 85

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Engineering at Meta

AUGUST 21, 2024

Llama 3 is Meta’s most capable openly-available LLM to date and the recently-released Llama 3.1 will enable new workflows, such as synthetic data generation and model distillation with unmatched flexibility, control, and state-of-the-art capabilities that rival the best closed source models. At AI Infra @ Scale 2024 , Meta engineers discussed every step of how we built and brought Llama 3 to life, from data and training to inference.

Software Engineer 94

Software Engineer Software Engineering Engineering Cloud 94

KDnuggets

AUGUST 20, 2024

Looking to make your data visuals stand out? Check out these five tips for effective data visualization.

Data 130

Data Data Science 130

Edureka

AUGUST 18, 2024

In the 21st century, Full Stack Web Development has undoubtedly transformed the internet. Notably, it is the driving force behind all the sites we see today, as well as mobile applications installed on our devices. Therefore, To succeed in web development, it’s important for anyone hoping to make a career out of it to understand and excel in the full stack developer skills required according to the current market trend.

Programming Language 98

Programming Language Java Non-relational Database Database 98

Rock the JVM

AUGUST 22, 2024

By Riccardo Cardin At Rock the JVM, we deeply understand the power of the Kotlin Arrow library and the Raise DSL, and we’ve previously shared our insights in our article on Functional Error Handling in Kotlin, Part 3: The Raise DSL. Now, we’re ready to introduce the crucial topic of testing applications that use the Raise DSL. To fully grasp the concepts we’ll be discussing, it’s essential to have a solid understanding of Kotlin, Arrow library, and the Raise DSL.

Portfolio 81

Portfolio Coding Programming Scala 81

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Engineering at Meta

AUGUST 22, 2024

In this talk from AI Infra @ Scale 2024 , Joel Colburn, a software engineer at Meta, technical lead Junqiang Lan, and software engineer Jack Montgomery discuss the second generation of MTIA , Meta’s in-house training and inference accelerator. They cover the co-design process behind building the second generation of Meta’s first-ever custom silicon for AI workloads, including the PyTorch software ecosystem, and the model architectures for Meta’s key applications.

Designing 82

Designing Software Engineer Software Engineering Utilities 82

KDnuggets

AUGUST 20, 2024

This article explains the basics of time series analysis. Learn to prepare your data and visualize trends in R.

Data 123

Data Programming 123

Cloudera

AUGUST 20, 2024

By now, every organization, regardless of industry, has at least explored the use of AI, if not already embraced it. In today’s market, the AI imperative is firmly here, and failing to act quickly could mean getting left behind. But even as adoption soars, struggles remain, and scalability continues to be a major issue. Organizations are quick to adopt AI, but getting it established across the organization brings a unique set of challenges that come into play.

Government 82

Government Data Lake Data Governance Data Architecture 82

Towards Data Science

AUGUST 23, 2024

Expert techniques to elevate your analysis Continue reading on Towards Data Science »

Data Science 75

Data Science SQL Data Data Engineering 75

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Snowflake

AUGUST 22, 2024

Uncertainty is the new norm for today’s brand advertisers and advertising agencies. Google has once again changed its stance on third-party cookies, keeping them for at least the time being. With the future of cookies still uncertain, many advertisers have already been preparing to use alternative targeting strategies, such as first-party data and contextual targeting.

Entertainment 75

Entertainment Media Algorithm Data Collection 75

KDnuggets

AUGUST 19, 2024

Learn how to harness Hugging Face pre-trained language models to classify the emotions behind tweets.

115

115

databricks

AUGUST 21, 2024

Special thanks to David Gray @Epsilon, Tanishq Bhalla @HealthVerity, Itai Weiss @ Nimble, JB Kole @ Mostly.ai for their valuable insights and contributions.

Data 75

Data 75

Towards Data Science

AUGUST 21, 2024

Getting to the bottom of what structuring your data responsibly really means Continue reading on Towards Data Science »

Data Science 75

Data Science Data Structured Data Data Architecture 75

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer