2021/09/02 by Rhys E. Green, Rhys Green, Green, Rhys +4
Computer Science · Mathematics · Psychology · #Adversarial Robustness in Machine Learning #Artificial intelligence #Calibration #Computer science #Confidence distribution #Confidence interval #Econometrics #Estimator #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Low Confidence #Machine Learning (cs.LG) #Machine Learning and Data Classification #Machine learning #Mathematics #Point (geometry) #Point estimation #Psychology #Statistics #Trustworthiness #Uncertainty quantification #cs.LG
paper · pdf · doi:10.48550/arxiv.2109.01531
published in arXiv (Cornell University) (Cornell University)
arxiv created 2021/09/02 · openalex publication_date 2021/09/02 · arxiv updated 2021/09/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Reliable Confidence Estimates are hugely important for any machine learning model to be truly useful. In this paper, we argue that any confidence estimates based upon standard machine learning point prediction algorithms are fundamentally flawed and under situations with a large amount of epistemic uncertainty are likely to be untrustworthy. To address these issues, we present MACEst, a Model Agnostic Confidence Estimator, which provides reliable and trustworthy confidence estimates. The algorithm differs from current methods by estimating confidence independently as a local quantity which explicitly accounts for both aleatoric and epistemic uncertainty. This approach differs from standard calibration methods that use a global point prediction model as a starting point for the confidence estimate.