Selective Classification Can Magnify Disparities Across Groups

2020/10/27 by Erik Jones, Jones, Erik, Shiori Sagawa +7 · 3 citations
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2010.14134

Published at the International Conference on Learning Representations (ICLR) 2021

openalex publication_date 2020/10/27 · arxiv created 2021/04/14 · arxiv updated 2021/04/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even decrease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model's confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies a property we call left-log-concavity. Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Motivated by our analysis, we train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.

Citations

• Equality of Opportunity in Supervised Learning
• Direct Uncertainty Prediction for Medical Second Opinions
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• Inherent Trade-Offs in the Fair Determination of Risk Scores
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Consistent Estimators for Learning to Defer to an Expert
• Noise or Signal: The Role of Image Backgrounds in Object Recognition
• WILDS: A Benchmark of in-the-Wild Distribution Shifts

Cited by

• What Does It Take to Build a Performant Selective Classifier?
• Unequal Uncertainty: Rethinking Algorithmic Interventions for Mitigating Discrimination from AI
• Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention
• The challenge of uncertainty quantification of large language models in medicine

Related

• Selective Ensembles for Consistent Predictions
• What Does It Take to Build a Performant Selective Classifier?
• Selective Regression Under Fairness Criteria
• Selective Classification via One-Sided Prediction
• Calibrated Selective Classification