KDnuggets

AUGUST 24, 2022

This blog post introduces seven techniques that are commonly applied in domains like intrusion detection or real-time bidding, because the datasets are often extremely imbalanced.

Datasets 160

Datasets Data Machine Learning 160

Simon Späti

AUGUST 25, 2022

Image by Rachel Claire on Pexels Ever wanted or been asked to build an open-source Data Lake offloading data for analytics? Asked yourself what components and features would that include. Didn’t know the difference between a Data Lakehouse and a Data Warehouse? Or you just wanted to govern your hundreds to thousands of files and have more database-like features but don’t know how?

Data Lake 130

Data Lake Data Warehouse Government Data 130

Data Engineering Podcast

AUGUST 21, 2022

Summary Data has permeated every aspect of our lives and the products that we interact with. As a result, end users and customers have come to expect interactions and updates with services and analytics to be fast and up to date. In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences.

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Cloudera

AUGUST 24, 2022

Cloudera Machine Learning (CML) is a cloud-native and hybrid-friendly machine learning platform. It unifies self-service data science and data engineering in a single, portable service as part of an enterprise data cloud for multi-function analytics on data anywhere. CML empowers organizations to build and deploy machine learning and AI capabilities for business at scale, efficiently and securely, anywhere they want.

Machine Learning 83

Machine Learning Building Metadata Programming Language 83

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

AUGUST 25, 2022

MLFlow is a tool to manage the end-to-end lifecycle of a Machine Learning model. Likewise, the installation and configuration of an MLFlow service is addressed and examples are added on how to generate and share projects with MLFlow in Layer.

Machine Learning 144

Machine Learning Project Management 144

Netflix Tech

AUGUST 24, 2022

by Ehtsham Elahi with James McInerney , Nathan Kallus , Dario Garcia Garcia and Justin Basilico Introduction This writeup is about using reinforcement learning to construct an optimal list of recommendations when the user has a finite time budget to make a decision from the list of recommendations. Working within the time budget introduces an extra resource constraint for the recommender system.

Algorithm 75

Algorithm Systems Datasets Architecture 75

Confluent

AUGUST 25, 2022

G2 named Confluent the the event stream processing industry leader for top-rated performance, reliability, ease of use, integration APIs, data modeling features, and more.

Process 64

Process Data 64

KDnuggets

AUGUST 23, 2022

This tutorial will explore four scenarios in which you can apply different transformations to all DataFrame columns.

Python 144

Python Data Data Science 144

Rockset

AUGUST 26, 2022

The American dairy industry is a mighty one. America’s 32,000 dairy farmers not only produce the most milk in the world , they are also the most efficient, producing 23 thousand pounds of milk per cow per year — almost 20 times the weight of an average (1,200 pound) dairy cow. For their genetically strong herds, healthy cows, high yields, even increasingly green operations , farmers can credit both agricultural science as well as data science.

IT 52

IT Food Aggregated Data Data Lake 52

Teradata

AUGUST 26, 2022

Is insurance immune from the surge in data-driven applications in other industries? Of course not, but why has there been such a slow uptake in data resources?

Insurance 52

Insurance IT Data 52

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Confluent

AUGUST 23, 2022

The VP of Engineering at Confluent India shares how the team builds innovative, modern data solutions while instilling a humble, open work culture where employees thrive.

Data Solutions 64

Data Solutions Engineering Building Data 64

KDnuggets

AUGUST 22, 2022

Write a single line of code to clean and process the data for analytics and machine learning tasks.

Data Process 138

Data Process Process Machine Learning Data 138

U-Next

AUGUST 26, 2022

Introduction. R Programming Language: What Is It? R is available as an open language of programming for statistical computing and data analytics, and R often has a command-line API. R is accessible on popular operating systems, including Pc, Linux, and macintosh. The newest cutting-edge technology is the R programming language. The R Research Core Group is presently carrying out its research.

R (Programming) 52

R (Programming) Programming Language Algorithm Programming 52

Monte Carlo

AUGUST 25, 2022

“How do we scale data observability with Monte Carlo?” I’ve heard this from hundreds of new customers. They’re excited about all that data observability can do for them, but like with any new software, they want prescriptive guidance. “In the ‘Crawl → Walk → Run’ of software adoption, what’s the quickest way for my team to start crawling?” If you’re a data team of 5-15 engineers or analysts, I recommend building healthy data observability muscles using our end-to-end, out-of-the-box monitors , a

Datasets 52

Datasets BI Data SQL 52

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Confluent

AUGUST 26, 2022

Full introduction to Confluent Cloud networking: security, setup and configuration, cost considerations, and which networking option to choose for your architecture.

Cloud 59

Cloud Architecture 59

KDnuggets

AUGUST 22, 2022

Learn Python by doing Python. Check out this free project-based course to quickly learn how to program in the high-demand language.

Python 138

Python Project Coding Programming 138

Palantir

AUGUST 25, 2022

The Palantir Incident Response team addresses the highest-priority issues across our platforms — Foundry, Gotham, and Apollo — ensuring they continue to support mission-critical work around the world. Essentially, the team’s core mandate is to respond when things go wrong. More broadly, Incident Response focuses on business continuity while adapting to an ever-expanding feature set as development teams across Palantir continuously add new capabilities and enhancements.

Engineering 52

Engineering Management Python Coding 52

Wolt

AUGUST 24, 2022

Here at Wolt we truly love open-source software. We’re a fast-growing company, building the rocket ship while riding it to allow our business to scale. This wouldn’t be possible without standing on the shoulders of giant open-source projects. Almost our whole tech stack is based on open-source software, most notably on the data engineering side.

Data Engineering 52

Data Engineering Data Engineer Engineering Project 52

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Monte Carlo

AUGUST 24, 2022

What do Daniel Kahneman, the Nobel Prize-winning psychologist, economist, and author of Thinking, Fast and Slow , and Nate Silver, founder and editor-in-chief of opinion poll analysis website FiveThirtyEight , have in common? Not only are they two of the most interesting voices in data, but they’re speaking at IMPACT: The Data Observability Summit , from October 25-26, 2022.

Data Architect 52

Data Architect Data Warehouse Architecture Data 52

KDnuggets

AUGUST 22, 2022

Hyperparameter tuning is important for algorithms. It improves their overall performance of a machine learning model and is set before the learning process and happens outside of the model.

Machine Learning 126

Machine Learning Algorithm Process IT 126

dbt Developer Hub

AUGUST 24, 2022

Editors note - this post assumes working knowledge of dbt Package development. For an introduction to dbt Packages check out So You Want to Build a dbt Package. It’s important to be able to test any dbt Project, but it’s even more important to make sure you have robust testing if you are developing a dbt Package. I love dbt Packages, because it makes it easy to extend dbt’s functionality and create reusable analytics resources.

Utilities 52

Utilities Project Building Coding 52

U-Next

AUGUST 23, 2022

Introduction. If the results of the assessment of the information are displayed in the form of information representation, all the outstanding purpose-oriented corporate judgments become simple to pursue. Additionally, having all statistics, infographics, graphs, etc., on one dashboard makes it easier to foresee insights. Tableau serves as a visual framework for business intelligence and analytics, assisting users in watching, observing, comprehending, and making choices with various data types.

BI 52

BI Amazon Web Services Database Business Intelligence 52

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Monte Carlo

AUGUST 22, 2022

Across both public and private sectors, more organizations are adopting a “data-driven” mindset—or, at least, data-driven messaging. But in reality, most aren’t prepared for the reality of what it takes to truly make decisions based on data. Teams have to be aligned about what data is used and how decisions are made. Data has to be accessible and available to the right decision-makers at the right time.

Building 52

Building Data Consulting Software Engineer 52

KDnuggets

AUGUST 23, 2022

How to Perform Motion Detection Using Python • The Complete Collection of Data Science Projects – Part 2 • Free AI for Beginners Course • Decision Tree Algorithm, Explained • What Does ETL Have to Do with Machine Learning?

Python 123

Python Algorithm Machine Learning Data Science 123

dbt Developer Hub

AUGUST 23, 2022

Those who have been building data warehouses for a long time have undoubtedly encountered the challenge of building surrogate keys on their data models. Having a column that uniquely represents each entity helps ensure your data model is complete, does not contain duplicates, and able to join across different data models in your warehouse. Sometimes, we are lucky enough to have data sources with these keys built right in — Shopify data synced via their API, for example, has easy-to-use keys on a

Datasets 52

Datasets Metadata SQL Data Warehouse 52

AltexSoft

AUGUST 22, 2022

The landscape of enterprise data is fragmented. According to Flexera’s 2022 State of the Cloud Report , 89 percent of respondents have a multi-cloud strategy with 80 percent having a hybrid cloud approach in place. Organizations have data stored in public and private clouds, as well as in various on-premises data repositories. How organizations embrace multi-cloud.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

U-Next

AUGUST 26, 2022

Introduction. The communications system is growing quickly in the modern world. To increase organizational productivity, organizations are turning digital. Datasets are growing increasingly complicated due to an increase in the volume of data produced on the web. Searching in Data Structure enables the efficient retrieval of individual elements from a collection, such as a specific record from a database.

Algorithm 40

Algorithm Data Utilities Data Science 40

KDnuggets

AUGUST 25, 2022

Is data science for you? And if it is, how can you use it to grow your business?

Data Science 120

Data Science Data IT 120

Propel Data

AUGUST 24, 2022

With Snowflake, it’s possible to build an enterprise data warehouse (EDW), an operational data store (ODS), or a team-specific data mart.

Data Warehouse 40

Data Warehouse Data Analytics Data Building 40

KDnuggets

AUGUST 25, 2022

Build the essential technical, analytical, and leadership skills needed for careers in today's data-driven world in Northwestern’s Master of Science in Data Science program.

Data Science 115

Data Science Data Programming Building 115

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer