Confidence Calibration for Convolutional Neural Networks Using Structured Dropout

2019/06/23 by Zhilu Zhang, Zhang, Zhilu, Adrian V. Dalca +3 · 2 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1906.09551

arxiv created 2019/06/23 · openalex publication_date 2019/06/23 · arxiv updated 2019/06/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, confidence estimates computed with a naive implementation of dropout can be poorly calibrated, particularly when using convolutional networks. In this paper, through the lens of ensemble learning, we associate calibration error with the correlation between the models sampled with dropout. Motivated by this, we explore the use of structured dropout to promote model diversity and improve confidence calibration. We use the SVHN, CIFAR-10 and CIFAR-100 datasets to empirically compare model diversity and confidence errors obtained using various dropout techniques. We also show the merit of structured dropout in a Bayesian active learning application.

Citations

• Variational Dropout and the Local Reparameterization Trick
• A Practical Bayesian Framework for Backpropagation Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• The Description Length of Deep Learning Models
• DropBlock: A regularization method for convolutional networks
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking
• Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems

Related

• Dropout Distillation for Efficiently Estimating Model Confidence
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferences
• Dropout Injection at Test Time for Post Hoc Uncertainty Quantification in Neural Networks
• Novel Uncertainty Framework for Deep Learning Ensembles