Towards calibrated and scalable uncertainty representations for neural networks

2019/10/28 by Nabeel Seedat, Christopher Kanan, Seedat, Nabeel +1 · 1 citation
Computer Science · Physics and Astronomy · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.1911.00104

openalex publication_date 2019/10/28 · openalex created_date 2019/11/22 · openalex updated_date 2026/07/28

Abstract

For many applications it is critical to know the uncertainty of a neural network's predictions. While a variety of neural network parameter estimation methods have been proposed for uncertainty estimation, they have not been rigorously compared across uncertainty measures. We assess four of these parameter estimation methods to calibrate uncertainty estimation using four different uncertainty measures: entropy, mutual information, aleatoric uncertainty and epistemic uncertainty. We evaluate the calibration of these parameter estimation methods using expected calibration error. Additionally, we propose a novel method of neural network parameter estimation called RECAST, which combines cosine annealing with warm restarts with Stochastic Gradient Langevin Dynamics, capturing more diverse parameter distributions. When benchmarked against mutilated image data, we show that RECAST is well-calibrated and when combined with predictive entropy and epistemic uncertainty it offers the best calibrated measure of uncertainty when compared to recent methods.

Citations

• ImageNet classification with deep convolutional neural networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Sampling-based Bayesian Inference with gradient uncertainty
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• A Parsimonious Tour of Bayesian Model Uncertainty
• Weight Uncertainty in Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Cited by

• Aleatoric Uncertainty Medical Image Segmentation Estimation via Flow Matching

Related

• A Survey of Uncertainty in Deep Neural Networks
• Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off
• Improving model calibration with accuracy versus uncertainty optimization
• A survey of uncertainty in deep neural networks
• Dense Uncertainty Estimation