2025/01/31 by Eugène Berta, Berta, Eugène, David Holzmüller +6 · 4 voices · 5 citations
Computer Science · Mathematics · #Algorithm #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Calibration #Computer science #Early stopping #Error detection and correction #Explainable Artificial Intelligence (XAI) #Hyperparameter #Machine Learning and Data Classification #Machine learning #Mathematics #Metric (unit) #Overconfidence effect #Performance metric #Probabilistic logic #Statistics #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2501.19195
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/01/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses, such as cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we present a novel variational formulation of the calibration-refinement decomposition that sheds new light on post-hoc calibration, and enables rapid estimation of the different terms. Equipped with this new perspective, we provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training. Selecting the best epoch based on validation loss thus leads to a compromise point that is suboptimal for both terms. To address this, we propose minimizing refinement error only during training (Refine,...), before minimizing calibration error post hoc, using standard techniques (...then Calibrate). Our method integrates seamlessly with any classifier and consistently improves performance across diverse classification tasks.