Recalibrating binary probabilistic classifiers

2025/05/25 by Tasche, Dirk
#68T09 #91G40 #FOS: Computer and information sciences #FOS: Economics and business #G.3 #I.5.1 #Machine Learning (cs.LG) #Risk Management (q-fin.RM)

paper · doi:10.48550/arxiv.2505.19068

Abstract

Recalibration of binary probabilistic classifiers to a target prior probability is an important task in areas like credit risk management. We analyse methods for recalibration from a distribution shift perspective. Distribution shift assumptions linked to the area under the curve (AUC) of a probabilistic classifier are found to be useful for the design of meaningful recalibration methods. Two new methods called parametric covariate shift with posterior drift (CSPD) and ROC-based quasi moment matching (QMM) are proposed and tested together with some other methods in an example setting. The outcomes of the test suggest that the QMM methods discussed in the paper can provide appropriately conservative results in evaluations with concave functionals like for instance risk weights functions for credit risk.

Citations

• On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift
• Cautious Calibration in Binary Classification
• Domain Adaptation with Factorizable Joint Shift
• Evaluating model calibration in classification
• A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation
• Detecting and Correcting for Label Shift with Black Box Predictors
• Estimation of inter-sector asset correlations
• Estimating discriminatory power and PD curves when the number of defaults is small
• An introduction to ROC analysis
• Adjusting the Outputs of a Classifier to New aPrioriProbabilities: A Simple Procedure

Related

• Calibration of Machine Learning Classifiers for Probability of Default Modelling
• Minimum-Risk Recalibration of Classifiers
• From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration
• Credit scoring using neural networks and SURE posterior probability calibration
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics