KDnuggets

SEPTEMBER 28, 2022

We explain how choosing a small, representative dataset from a large population can improve model training reliability.

Datasets 160

Datasets Machine Learning 160

Simon Späti

SEPTEMBER 30, 2022

Image by Rachel Claire on Pexels Ever wanted or been asked to build an open-source Data Lake offloading data for analytics? Asked yourself what components and features would that include. Didn’t know the difference between a Data Lakehouse and a Data Warehouse? Or you just wanted to govern your hundreds to thousands of files and have more database-like features but don’t know how?

Data Lake 130

Data Lake Data Warehouse Government Data 130

Marc Lamberti

SEPTEMBER 18, 2022

Airflow Taskflow is a new way of writing DAGs at ease. As you will see, you need to write fewer lines than before to obtain the same DAG. That helps to make DAGs easier to build, read, and maintain. The Taskflow API has three main aspects: XCOM Args, Decorator, and XCOM backends. In this tutorial, you will learn what the Taskflow API is, why it is crucial for you, and how to create your DAGs.

SQL 130

SQL Python Coding Accessibility 130

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data.

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Confluent

SEPTEMBER 14, 2022

How gaming enterprises like Sony and Big Fish Games use Apache Kafka®, Confluent, and ksqlDB’s data streaming technologies for the best in-game experience, ROI, and real-time capabilities.

Kafka 124

Kafka Technology Data 124

Cloudera

SEPTEMBER 7, 2022

We are now well into 2022 and the megatrends that drove the last decade in data — The Apache Software Foundation as a primary innovation vehicle for big data, the arrival of cloud computing, and the debut of cheap distributed storage — have now converged and offer clear patterns for competitive advantage for vendors and value for customers. Cloudera has been parlaying those patterns into clear wins for the community at large and, more importantly, streamlining the benefits of that innovation to

Big Data Ecosystem 118

Big Data Ecosystem Hadoop Big Data Architecture 118

Simon Späti

SEPTEMBER 29, 2022

A semantic layer is something we use every day. We build dashboards with yearly and monthly aggregations. We design dimensions for drilling down reports by region, product, or whatever metrics we are interested in. What has changed is that we no longer use a singular business intelligence tool; different teams use different visualizations (BI, notebooks, and embedded analytics).

BI 130

BI Business Intelligence Designing Building 130

U-Next

SEPTEMBER 27, 2022

Introduction . Cybersecurity or computer security and information security is the act of preventing theft, damage, loss, or unauthorized access to computers, networks, and data. As our interconnections grow, so do the chances for evil hackers to steal, destroy, or disrupt our lives. The increase in cybercrime has increased the demand for cybersecurity expertise.

Certification 116

Certification Consulting Computer Science Government 116

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Regardless of how data is being used, it is critical that the information is trusted. The practice of data reliability engineering has gained momentum recently to address that question. To help support the efforts of data teams the folks at Soda Data created the Soda Checks Language and the corresponding Soda Core utility that acts on this new DSL.

Building 100

Building Metadata MongoDB MySQL 100

Confluent

SEPTEMBER 20, 2022

Microservices have numerous benefits, but data silos are incredibly challenging. Learn how Kafka Connect and CDC provide real-time database synchronization, bridging data silos between all microservice applications.

Kafka 122

Kafka Database Data 122

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Cloudera

SEPTEMBER 23, 2022

In a recent blog, Cloudera Chief Technology Officer Ram Venkatesh described the evolution of a data lakehouse, as well as the benefits of using an open data lakehouse, especially the open Cloudera Data Platform (CDP). If you missed it, you can read up about it here. Modern data lakehouses are typically deployed in the cloud. Cloud computing brings several distinct advantages that are core to the lakehouse value proposition.

Data Governance 114

Data Governance Government Amazon Web Services Cloud Computing 114

KDnuggets

SEPTEMBER 22, 2022

How to increase your chances of avoiding the mistake.

Data 160

Data Data Science 160

Simon Späti

SEPTEMBER 29, 2022

A semantic layer is something we use every day. We build dashboards with yearly and monthly aggregations. We design dimensions for drilling down reports by region, product, or whatever metrics we are interested in. What has changed is that we no longer use a singular business intelligence tool; different teams use different visualizations (BI, notebooks, and embedded analytics).

BI 130

BI Business Intelligence Designing Building 130

Teradata

SEPTEMBER 30, 2022

With the release of VantageCloud Lake and ClearScape Analytics, Teradata brings a cloud-native architecture to extend the technical innovations and differentiators that Vantage is well known for.

Architecture 98

Architecture Cloud Data 98

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. With the increasing expecation for information to be instantly accessible, it drives the need for reliable change data capture. The team at Fivetran have recently introduced that functionality to power real-time data products.

Food 100

Food MongoDB MySQL Scala 100

Confluent

SEPTEMBER 28, 2022

Highlighting sessions on the power of our Confluent-Google partnership: multi-layer data security, real-time cloud data streaming and analytics, database modernization, and more.

Google Cloud 98

Google Cloud Cloud Data Security Database 98

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested. Imagine independently discovering rich new business insights from both structured and unstructured data working together, without having to beg for data sets to be made available.

Architecture 108

Architecture Metadata Machine Learning Unstructured Data 108

KDnuggets

SEPTEMBER 20, 2022

When building and optimizing your classification model, measuring how accurately it predicts your expected outcome is crucial. However, this metric alone is never the entire story, as it can still offer misleading results. That's where these additional performance evaluations come into play to help tease out more meaning from your model.

Building 160

Building IT Machine Learning 160

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5 years, its usage has increased, and Timestone is now also the priority queueing engine backing Conductor , our general-purpose workflow orchestration engine, and BDP Sch

Systems 89

Systems Metadata Media Kafka 89

Teradata

SEPTEMBER 1, 2022

Financial services organizations that exhibit true data literacy avoid bottlenecks and instead choose to build best in class solutions that meet current and future needs. Find out more.

Building 95

Building Data 95

Data Engineering Podcast

SEPTEMBER 18, 2022

Summary In order to improve efficiency in any business you must first know what is contributing to wasted effort or missed opportunities. When your business operates across multiple locations it becomes even more challenging and important to gain insights into how work is being done. In this episode Tommy Yionoulis shares his experiences working in the service and hospitality industries and how that led him to found OpsAnalitica, a platform for collecting and analyzing metrics on multi location

Hospitality 100

Hospitality Food MongoDB MySQL 100

Confluent

SEPTEMBER 21, 2022

A deep dive into how microservices work, why it’s the backbone of real-time applications, and how to build event-driven microservices applications with Python and Kafka.

Kafka 98

Kafka Python Building 98

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Cloudera

SEPTEMBER 22, 2022

Insurance carriers are always looking to improve operational efficiency. We’ve previously highlighted opportunities to improve digital claims processing with data and AI. In this post, I’ll explore opportunities to enhance risk assessment and underwriting, especially in personal lines and small and medium-sized enterprises. Underwriting is an area that can yield improvements by applying the old saying “work smarter, not harder.

Insurance 107

Insurance Medical Machine Learning Data 107

KDnuggets

SEPTEMBER 16, 2022

Why is Gradient Descent so important in Machine Learning? Learn more about this iterative optimization algorithm and how it is used to minimize a loss function.

Machine Learning 160

Machine Learning Algorithm IT 160

dbt Developer Hub

SEPTEMBER 7, 2022

If you’ve ever heard of Marie Kondo, you’ll know she has an incredibly soothing and meditative method to tidying up physical spaces. Her KonMari Method is about categorizing, discarding unnecessary items, and building a sustainable system for keeping stuff. As an analytics engineer at your company, doesn’t that last sentence describe your job perfectly?!

Designing 80

Designing Data Coding SQL 80

U-Next

SEPTEMBER 30, 2022

Show, don’t tell is what people tell writers and screenwriters, but this is practically applicable to aspirants who want to land their dream jobs as well. Apart from your university degree and professional certifications, what adds compelling weightage to your candidacy is a solid portfolio. . A portfolio is like your business card and regardless of whether you are a fresher or someone experienced, moving up the corporate ladder, a portfolio is what will ensure you a job, that higher paycheck a

Portfolio 52

Portfolio Recruitment Certification Project 52

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Data Engineering Podcast

SEPTEMBER 18, 2022

Summary There is a constant tension in business data between growing siloes, and breaking them down. Even when a tool is designed to integrate information as a guard against data isolation, it can easily become a silo of its own, where you have to make a point of using it to seek out information. In order to help distribute critical context about data assets and their status into the locations where work is being done Nicholas Freund co-founded Workstream.

Building 100

Building Metadata MongoDB MySQL 100

Confluent

SEPTEMBER 12, 2022

How banks and finance companies use Confluent to transform their digital systems with event-driven architecture, real-time payment processing, fraud detection, and analytics.

Banking 95

Banking Finance Architecture Data 95

Cloudera

SEPTEMBER 19, 2022

The identity team at Cloudera has been working to add the System for Cross-domain Identity Management (SCIM) support to Cloudera Data Platform (CDP) and we’re happy to announce the general availability of SCIM on Azure Active Directory! In Part One we discussed: CDP SCIM Support for Active Directory, which discusses the core elements of CDP’s SCIM support for Azure AD.

Systems 107

Systems Management Accessibility Accessible 107

KDnuggets

SEPTEMBER 13, 2022

This article will go over the top 5 data science skills that pay you and 5 that don’t.

Data Science 160

Data Science Data 160

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems