KDnuggets

FEBRUARY 25, 2022

Pick your visualizations strategically. They need to tell a story.

Data 160

Data Data Science 160

Start Data Engineering

FEBRUARY 22, 2022

1. Introduction 2. CI 3. Sample project: Data testing with Github Actions 3.1. Prerequisites 3.2. Project overview 3.3. Automating data tests with Github Actions 4. Conclusion 5. Further reading 1. Introduction Automated testing is crucial for ensuring that your code is bug-free and avoiding regressions. If you are wondering How can data tests be integrated into a CI (Continuous Integration) pipeline?

Data 130

Data Project Coding Systems 130

Marc Lamberti

FEBRUARY 21, 2022

Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice way of generating DAGs comes at the price of higher complexity and subtle tricky things that you must know.

Python 130

Python Metadata Coding Process 130

Confluent

FEBRUARY 24, 2022

A few years ago I helped build an event-driven system for gym bookings. The pitch was that we were building a better experience for both the gym members booking different […].

Systems 124

Systems Building Data 124

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

FEBRUARY 25, 2022

This blog post aims to describe the vanishing gradient problem and explain how use of the sigmoid function resulted in it.

IT 160

IT Machine Learning 160

Cloudera

FEBRUARY 22, 2022

Over the past decade, the successful deployment of large scale data platforms at our customers has acted as a big data flywheel driving demand to bring in even more data, apply more sophisticated analytics, and on-board many new data practitioners from business analysts to data scientists. This unprecedented level of big data workloads hasn’t come without its fair share of challenges.

Metadata 110

Metadata Datasets BI SQL 110

ProjectPro

FEBRUARY 25, 2022

What is a Machine Learning Pipeline? A machine learning pipeline helps automate machine learning workflows by processing and integrating data sets into a model, which can then be evaluated and delivered. A well-built pipeline helps in the flexibility of the model implementation. A pipeline in machine learning is a technical infrastructure that allows an organization to organize and automate machine learning operations.

Machine Learning 52

Machine Learning Building Amazon Web Services AWS 52

KDnuggets

FEBRUARY 22, 2022

Machine Learning as a technology, ensures that our current gadgets and their software get smarter by the day. Here are the algorithms that you ought to know about to understand Machine Learning’s varied and extensive functionalities and their effectiveness.

Machine Learning 160

Machine Learning Algorithm Technology 160

Cloudera

FEBRUARY 23, 2022

The telecommunications industry has been doing well since the pandemic started (not that many would notice). Revenues have remained relatively stable, while consumption has gone up, as virtual engagement has become the primary mode of operations for many businesses (and families!) In the mean-time, digital transformation has been accelerating both as a means to respond to the pandemic, and as a mechanism to drive costs down further, allowing for margin growth.

Telecommunication 95

Telecommunication Cloud Media Government 95

Data Engineering Podcast

FEBRUARY 20, 2022

Summary Python has grown to be one of the top languages used for all aspects of data, from collection and cleaning, to analysis and machine learning. Along with that growth has come an explosion of tools and engines that help power these workflows, which introduces a great deal of complexity when scaling from single machines and exploratory development to massively parallel distributed computation.

Python 100

Python Data Process IT Building 100

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

ProjectPro

FEBRUARY 25, 2022

When the world was under lockdown and movement was restricted to an absolute emergency- millions were introduced to the world of online shopping. The convenience of online shopping helped e-commerce platforms record historic sales. While that happened, it is no surprise that the rate of online financial fraud also increased incredibly. Online fraud cases using credit and debit cards saw a historic upsurge of 225 percent during the COVID-19 pandemic in 2020 as compared to 2019.

Machine Learning 52

Machine Learning Project Algorithm Datasets 52

KDnuggets

FEBRUARY 23, 2022

This article outlines some of the most common design patterns encountered when creating successful Machine Learning solutions.

Machine Learning 157

Machine Learning Designing 157

Cloudera

FEBRUARY 24, 2022

In the latest installment of the EMEA Influential Women in Data webinar series, we welcomed Shirley Collie, Chief Health Analytics Actuary at Discovery Health to discuss everything from how the pandemic has impacted working, to the opportunities within data, and the importance of intentionality. A data-driven organization. Shirley knows better than most about the impact that COVID 19 has had on the world.

Healthcare 85

Healthcare Algorithm IT Management 85

Hepta Analytics

FEBRUARY 24, 2022

Week 3 was about data warehousing, working on the data that was ingested in the week 2. We will take the already ingested data and create an external table from it and optimize the performance of queries through partitioning and clustering. Then automate the whole process using airflow. There are two systems types when dealing with data: Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP).

Data Warehouse 52

Data Warehouse Data Engineering Data Engineer Engineering 52

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

ProjectPro

FEBRUARY 25, 2022

Read this article to find the right resources for learning MLOps. The blog starts with an introduction to MLOps, skills required to become an MLOps engineer, and then lays out an MLOps learning path for beginners. MLOps is an acronym that represents the combination of Machine-Learning(ML) and Operations. It is a beautiful technique for implementing data science projects that allow businesses to increase their projects’ efficiency minimize the risk of introducing machine learning, artificia

Deep Learning 52

Deep Learning Algorithm Machine Learning Data Science 52

KDnuggets

FEBRUARY 21, 2022

A collection of cheat sheets that will help you prepare for a technical interview on Data Structures & Algorithms, Machine learning, Deep Learning, Natural Language Processing, Data Engineering, Web Frameworks.

Data Science 153

Data Science Deep Learning Algorithm Machine Learning 153

Rockset

FEBRUARY 24, 2022

Event-based architectures have been gaining popularity for some time. With increased adoption has come a flood of options for aggregating and analyzing events. Which databases are optimized for ingesting streaming events and analyzing them in real time? The answer is complex, nuanced and heavily dependent on the precise problem being solved. This post is intended to help anyone seeking to make a selection from a difficult to understand landscape.

AWS 52

AWS Amazon Web Services Kafka SQL 52

Preset

FEBRUARY 23, 2022

This article walks you through a potential approach to monitor your Superset usage directly within Superset leveraging the internal metadata database.

Metadata 52

Metadata Database 52

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

ProjectPro

FEBRUARY 24, 2022

Working with audio data has been a relatively less widespread and explored problem in machine learning. In most cases, benchmarks for the latest seminal work in deep learning are measured on text and image data performances. Moreover, the most significant advances in deep learning are found in models that work with text and images. Amidst this, speech and audio, an equally important type of data, often gets overlooked.

Deep Learning 52

Deep Learning Project Datasets Machine Learning 52

KDnuggets

FEBRUARY 24, 2022

Object-relational mapping, or ORM, is a technique that allows you to interact with databases using the object-oriented paradigm of the programming language of your choosing. How is that different from structured query language, though, and when do you use them?

SQL 152

SQL Programming Language Database Programming 152

Confluent

FEBRUARY 23, 2022

In many ways, Storyblocks’ technical journey has mirrored that of most other startups and disruptors: Start small and as simple as possible (i.e., with a PHP monolith) Watch the company […].

Cloud 52

Cloud 52

Pipeline Data Engineering

FEBRUARY 22, 2022

Data engineering salon. News and interesting reads about the world of data. We’ve only scratched the surface of the full potential for the data warehouse Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank Why I think the data warehouse will become the control centre for modern companies Git, SQL, CLI Vicki Boykis, Machine Learning Engineer, Automattic I’ve narrowed it down to three basic tools.

Data Warehouse 52

Data Warehouse Banking Metadata Data Science 52

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

ProjectPro

FEBRUARY 21, 2022

Learn PySpark Joins in a single go! From the various type of PySpark joins to their syntax and PySpark join example, this blog has it all for you. Without much ado, let’s dive right into the concepts. Table of Contents Why are PySpark Joins Important for Data Analytics? PySpark Joins- Types of Joins with Examples General Syntax for PySpark Join- PySpark Inner Join PySpark Left Join / PySpark Left Outer Join PySpark Right Join/ PySpark Right Outer Join PySpark Full Outer Join PySpark Left S

Data Analysis 52

Data Analysis Finance Datasets Big Data 52

KDnuggets

FEBRUARY 22, 2022

Learn how to optimize your data science workflow in a few lines of code.

Data Science 152

Data Science Python Project Data 152

Preset

FEBRUARY 21, 2022

CrateDB is a distributed SQL database that excels at IoT and Time Series data workflows. In this post, we'll showcase how CrateDB and Superset can be used together.

Data Workflow 52

Data Workflow SQL Database Data 52

AltexSoft

FEBRUARY 21, 2022

There has been a lot of buzz around data science, machine learning (ML), and artificial intelligence (AI) lately. As you may already know, to train a machine learning model, you need data. Lots of data, to be more precise. Lots of quality data, to be even more precise. To save you time, watch our 14-minute video on how data is prepared for machine learning.

Machine Learning 52

Machine Learning Algorithm Healthcare Medical 52

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

U-Next

FEBRUARY 21, 2022

Artificial Intelligence (AI) is not just making our lives convenient. It is empowering us with information and insights that have the potential to change the world for the better. With its application across diverse industries, market segments and real-world concerns, the role of AI is becoming increasingly inevitable by the day. This is to the extent that we see AI as a savior to some of the most plaguing concerns of humankind.

IT 52

IT Education Machine Learning 52

KDnuggets

FEBRUARY 25, 2022

Learn data analytics by taking the best YouTube courses. These courses will cover data analysis with Python, R, SQL, PowerBI, Tableau, Excel, and SPSS.

Data Analytics 150

Data Analytics SQL Data Analysis Python 150

ProjectPro

FEBRUARY 22, 2022

This article will teach you exciting SQL project ideas to develop data analysis skills. You will explore challenging problems that you can quickly solve with this simple query language. It doesn’t matter if you are a beginner or a professional at using SQL; our list of SQL database projects has one for you. Data, data, everywhere! Where’s the way to manage it?

Data Analysis 52

Data Analysis SQL Project Banking 52

KDnuggets

FEBRUARY 25, 2022

The data scientist salary - the past, the present, and a little bit of the future.

Data 149

Data 149

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer