KDnuggets

FEBRUARY 18, 2022

AI and machine learning can provide us with these tools. This guide will explore how we can use machine learning to label data.

Machine Learning 160

Machine Learning Data 160

Netflix Tech

FEBRUARY 18, 2022

By: Ankush Gulati , David Gevorkyan Additional credits: Michael Clark , Gokhan Ozer Intro Netflix has more than 220 million active members who perform a variety of actions throughout each session, ranging from renaming a profile to watching a title. Reacting to these actions in near real-time to keep the experience consistent across devices is critical for ensuring an optimal member experience.

Systems 133

Systems Architecture Portfolio Designing 133

Confluent

FEBRUARY 18, 2022

As data flows in and out of your Confluent Cloud clusters, it’s imperative to monitor their behavior. Bring Your Own Monitoring (BYOM) means you can configure an application performance monitoring […].

Cloud 119

Cloud Data 119

Cloudera

FEBRUARY 14, 2022

Note: This is part 2 of the Make the Leap New Year’s Resolution series. For part 1 please go here. When we introduced Cloudera Data Engineering (CDE) in the Public Cloud in 2020 it was a culmination of many years of working alongside companies as they deployed Apache Spark based ETL workloads at scale. We not only enabled Spark-on-Kubernetes but we built an ecosystem of tooling dedicated to the data engineers and practitioners from first-class job management API & CLI for dev-ops automatio

Data Engineering 113

Data Engineering Data Engineer Engineering Data Pipeline 113

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

KDnuggets

FEBRUARY 17, 2022

There's no free lunch in machine learning. So, determining which algorithm to use depends on many factors from the type of problem at hand to the type of output you are looking for. This guide offers several considerations to review when exploring the right ML approach for your dataset.

Machine Learning 160

Machine Learning Algorithm Datasets 160

Data Engineering Podcast

FEBRUARY 13, 2022

Summary Collecting, integrating, and activating data are all challenging activities. When that data pertains to your customers it can become even more complex. To simplify the work of managing the full flow of your customer data and keep you in full control the team at Rudderstack created their eponymous open source platform that allows you to work with first and third party data, as well as build and manage reverse ETL workflows.

Building 100

Building Hadoop Data Pipeline Metadata 100

Cloudera

FEBRUARY 17, 2022

CDP Private Cloud Base is an on-premises version of Cloudera Data Platform (CDP). This new product combines the best of Cloudera Enterprise Data Hub and Hortonworks Data Platform Enterprise along with new features and enhancements across the stack. This unified distribution is a scalable and customizable platform where you can securely run many types of workloads.

Cloud 101

Cloud Data Professional Services Metadata 101

KDnuggets

FEBRUARY 14, 2022

Calculus is the key to fully understanding how neural networks function. Go beyond a surface understanding of this mathematics discipline with these free course materials from MIT.

Deep Learning 160

Deep Learning Machine Learning 160

Data Engineering Podcast

FEBRUARY 13, 2022

Summary Streaming data sources are becoming more widely available as tools to handle their storage and distribution mature. However it is still a challenge to analyze this data as it arrives, while supporting integration with static data in a unified syntax. Deephaven is a project that was designed from the ground up to offer an intuitive way for you to bring your code to your data, whether it is streaming or static without having to know which is which.

Coding 100

Coding Engineering Data Pipeline Java 100

Pipeline Data Engineering

FEBRUARY 18, 2022

Data engineers have to wear many different hats at the same time: they are architects, designers, builders, maintainers, procurement and quality assurance — to just name a few. If you’d like to break into this profession, you need to prove that you can do all of the above, and more. One of the key assets you can use to do that is a data product that you’ve built with your own hands.

Building 52

Building Portfolio Kafka Machine Learning 52

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Cloudera

FEBRUARY 18, 2022

The pandemic has accelerated diversity of teams, remote working, and the way we work, but most of all, it has emphasised the necessity of soft skills in our leaders. Empathy stands out as a core skill that must be alive and nurtured within our teams if we are to achieve our desired outcomes in 2022 and beyond. This blog explores what empathy looks like in a business context, why it’s so important, and what we’re up to at Cloudera.

Banking 87

Banking Data Lake Technology Building 87

KDnuggets

FEBRUARY 16, 2022

In this article, I will walk you through how you can use your data science skills to land freelance gigs.

Data Science 159

Data Science Data 159

ProjectPro

FEBRUARY 18, 2022

The big data analytics market is expected to grow at a CAGR of 13.2 percent, reaching USD 549.73 billion in 2028. This indicates that more businesses will adopt the tools and methodologies useful in big data analytics, including implementing the ETL pipeline. Data engineers are in charge of developing data models, constructing data pipelines, and monitoring ETL (extract, transform, load).

Project 52

Project AWS Kafka Healthcare 52

DataKitchen

FEBRUARY 18, 2022

In this webinar, take a trip to DataOps 101 and learn the basics! The post DataOps For Beginners first appeared on DataKitchen.

52

52

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Cloudera

FEBRUARY 16, 2022

While it is a little dated, one amusing example that has been the source of countless internet memes is the famous, “is this a chihuahua or a muffin?” classification problem. Figure 01: Is this a chihuahua or a muffin? In this example, the Machine Learning (ML) model struggles to differentiate between a chihuahua and a muffin. The eyes and nose of a chihuahua, combined with the shape of its head and colour of its fur do look surprising like a muffin if we squint at the images in figure 01 above.

Machine Learning 85

Machine Learning Algorithm Government Metadata 85

KDnuggets

FEBRUARY 18, 2022

Check out this reasoned comparison of 2 critical machine learning algorithms to help you better make an informed decision.

Machine Learning 158

Machine Learning Algorithm 158

ProjectPro

FEBRUARY 17, 2022

Feature selection techniques are fundamental to predictive modeling tasks; one can not create predictive models without selecting the features correctly. What are these feature selection methods, and how are they used in building efficient predictive models? You will find out all the answers in this article. If you have ever baked a cake in your life or perhaps witnessed someone following a recipe to bake it, you must have noticed how crucial it is to precisely measure each ingredient's quantity

Machine Learning 52

Machine Learning Algorithm Datasets Banking 52

DataKitchen

FEBRUARY 18, 2022

DataOps is a discipline focused on the delivery of data faster, better, and cheaper to derive business value quickly. It closely follows the best practices of DevOps although the implementation of DataOps to data is nothing like DevOps to code. This paper will focus on providing a prescriptive approach in implementing a data pipeline using a DataOps discipline for data practitioners.

High Quality Data 52

High Quality Data Business Intelligence Data Pipeline Government 52

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This was used to test our setup. This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

KDnuggets

FEBRUARY 18, 2022

Octoparse 8.5 is now released with game-changing new features and major improvements.

152

152

ProjectPro

FEBRUARY 16, 2022

Early in 2020, when Myntra launched its visual product search for the first time, it created waves in e-commerce. With this new feature, the customers no longer had to spend hours searching for a dress similar to the one they came across randomly in an advertisement. All they had to do was take a picture/screenshot and upload it on Myntra; the app would automatically fetch outfits similar to the picture.

Architecture 52

Architecture Deep Learning Banking Datasets 52

Rockset

FEBRUARY 14, 2022

Fast analytics on fresh data is better than slow analytics on stale data. Fresh beats stale every time. Fast beats slow in every space. Time and time again, companies in a wide variety of industries have boosted revenue, increased productivity and cut costs by making the leap from batch analytics to real-time analytics. One of the perks of my job is getting to work every day with trailblazers of the real-time revolution, whether it is Doug Moore at construction SaaS provider Command Alkon , Carl

BI 52

BI Data Warehouse ETL Tools Data Lake 52

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

Preset

FEBRUARY 13, 2022

Apache Superset is the most popular open-source data exploration and visualization platform in the world. YugabyteDB is a distributed SQL database that works seamlessly using the standards PostgreSQL connector.

PostgreSQL 52

PostgreSQL SQL Database Data 52

KDnuggets

FEBRUARY 14, 2022

Give a boost to your career and learn job-ready machine learning skills by taking the best free online courses.

Machine Learning 132

Machine Learning 132

ProjectPro

FEBRUARY 16, 2022

Optical Character Recognition (OCR) has been used for decades across multiple sectors in the industry, such as banking, retail, healthcare, transportation, and manufacturing. With a tremendous increase in digitization in this 21st century, a.k.a Information age, OCR Python applications are witnessing huge demand. In fact, according to a recent survey, the market share of OCR will increase by 16.7% (compound annual growth rate) from 2021 to 2028 from 7.46 billion USD in 2020.

Python 52

Python Banking Data Science Transportation 52

Zalando Engineering

FEBRUARY 16, 2022

Persisted Queries Persisted Queries in GraphQL are like stored procedures in Databases. To know about the Apollo's way of automated persisted queries, please follow their documentation here. In Zalando, we took a different approach - to disable GraphQL in production. It might sound counterintuitive at first - we have a GraphQL service, but we disable GraphQL in production - why?

Database 40

Database Coding Engineering Designing 40

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Rockset

FEBRUARY 17, 2022

It’s the start of 2022 and a great time to look ahead and think about what changes we can expect in the coming months. If we’ve learned any lessons from the past, it’s that keeping ahead of the waves of change is one of the primary challenges of working in this industry. We asked thought leaders in our industry to ponder what they believe will be the new ideas that will influence or change the way we do things in the coming year.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

KDnuggets

FEBRUARY 17, 2022

Learn how to train, optimize, and build API with a few lines of code using AutoXGB.

Coding 123

Coding Building Machine Learning 123

KDnuggets

FEBRUARY 18, 2022

The best way to stop your enemy is to know your enemy. Here are four ways hackers are using data science - and how they can be stopped.

Data Science 116

Data Science Data 116

KDnuggets

FEBRUARY 17, 2022

Data Mesh in Action by Jacek Majchrzak, Sven Balnojan, and Marian Siwiak reveals how this new groundbreaking decentralized architecture looks for both small startups and large enterprises.

Architecture 112

Architecture Data 112

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer