MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator

2021/09/02 by Rhys E. Green, Green, Rhys, Matthew Rowe +3
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2109.01531

openalex publication_date 2021/09/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reliable Confidence Estimates are hugely important for any machine learning model to be truly useful. In this paper, we argue that any confidence estimates based upon standard machine learning point prediction algorithms are fundamentally flawed and under situations with a large amount of epistemic uncertainty are likely to be untrustworthy. To address these issues, we present MACEst, a Model Agnostic Confidence Estimator, which provides reliable and trustworthy confidence estimates. The algorithm differs from current methods by estimating confidence independently as a local quantity which explicitly accounts for both aleatoric and epistemic uncertainty. This approach differs from standard calibration methods that use a global point prediction model as a starting point for the confidence estimate.

Citations

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• To Trust Or Not To Trust A Classifier
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Related

• Trust, or Don't Predict: Introducing the CWSA Family for Confidence-Aware Model Evaluation
• Rethinking Uncertainty Evaluation in Large Language Models
• Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
• Useful Confidence Measures: Beyond the Max Score
• ConfTuner: Training Large Language Models to Express Their Confidence Verbally