Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs

2025/06/17 by Flores, Gerardo A., Julia Fukuyama, Smith, Alyssa H. +2
Medicine · #Artificial Intelligence (cs.AI) #Clinical practice guidelines implementation #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.14540

openalex publication_date 2025/06/17 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Machine learning-based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prioritize patient management decisions. However, widely used scoring rules, such as accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness to distributional shifts, and sensitivity to asymmetric error costs. In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for the uncertainty in class prevalences and domain-specific cost asymmetries often found in clinical settings. Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an adjusted variant of cross-entropy (log score) that averages cost-weighted performance over clinically relevant ranges of class balance. The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-world variations.

Citations

• Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration
• Weighted Brier Score -- an Overall Summary Measure for Risk Prediction Models with Clinical Utility Consideration
• Information-theoretic Generalization Analysis for Expected Calibration Error
• A Closer Look at AUROC and AUPRC under Class Imbalance
• Evaluating Probabilistic Classifiers: The Triptych
• Analysis and Comparison of Classification Metrics
• Notes on the H-measure of classifier performance
• Mitigating Bias in Calibration Error Estimation
• A Unified View of Label Shift Estimation
• Measuring Calibration in Deep Learning
• Regularized Learning for Domain Adaptation under Label Shifts
• Evaluating model calibration in classification
• The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric
• Learning Models with Uniform Performance via Distributionally Robust\n Optimization
• Detecting and Correcting for Label Shift with Black Box Predictors
• On Calibration of Modern Neural Networks
• Algorithmic decision making and the cost of fairness
• Inherent Trade-Offs in the Fair Determination of Risk Scores
• Inherent Trade-Offs in the Fair Determination of Risk Scores
• Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet\n Representations, and Forecast Rankings
• Learning Transferable Features with Deep Adaptation Networks
• Domain Generalization via Invariant Feature Representation
• A better Beta for the H measure of classification performance
• Threshold Choice Methods: the Missing Link
• Strictly Proper Scoring Rules, Prediction, and Estimation
• Using AUC and accuracy in evaluating learning algorithms
• Some Practical Issues of Experimental Design and Data Analysis in Radiological ROC Studies
• The meaning and use of the area under a receiver operating characteristic (ROC) curve.
• The area above the ordinal dominance graph and the area below the receiver operating characteristic graph
• A New Vector Partition of the Probability Score
• A unifying view on dataset shift in classification
• Adjusting the Outputs of a Classifier to New aPrioriProbabilities: A Simple Procedure
• Sample Selection Bias as a Specification Error
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

Related

• A Consequentialist Critique of Binary Classification Evaluation: Theory, Practice, and Tools
• Decision-Making under Miscalibration
• Analysis and Comparison of Classification Metrics
• Automatic Construction of Clinical Scoring Systems with LLM Agents
• Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare