A Skew-Sensitive Evaluation Framework for Imbalanced Data Classification

2020/10/12 by Min Du, Nesime Tatbul, Du, Min +17
Computer Science · Engineering · Health Professions · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare #Electricity Theft Detection Techniques #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2010.05995

openalex publication_date 2020/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes, making fair assessment of classifiers a challenging task. Metrics such as Balanced Accuracy are commonly used to evaluate a classifier's prediction performance under such scenarios. However, these metrics fall short when classes vary in importance. In this paper, we propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-art classifiers tested on real-world datasets from three different domains show the effectiveness of our framework - not only in evaluating and ranking classifiers, but also training them.

Related

• Review of Methods for Handling Class-Imbalanced in Classification Problems
• Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics
• On Model Evaluation under Non-constant Class Imbalance
• An Empirical Analysis of the Efficacy of Different Sampling Techniques for Imbalanced Classification
• Bad practices in evaluation methodology relevant to class-imbalanced problems