Dropout as a Bayesian Approximation: Appendix

2015/06/06 by Yarin Gal, Zoubin Ghahramani, Gal, Yarin +1 · 5 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (stat.ML) #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.1506.02157

openalex publication_date 2015/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitting. Our interpretation allows us to reason about uncertainty in deep learning, and allows the introduction of the Bayesian machinery into existing deep learning frameworks in a principled way. This document is an appendix for the main paper "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning" by Gal and Ghahramani, 2015.

Citations

• A Practical Bayesian Framework for Backpropagation Networks
• Variational Bayesian Inference with Stochastic Search
• ImageNet classification with deep convolutional neural networks

Cited by

• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Explainable Image Classification with Reduced Overconfidence for Tissue Characterisation
• That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation
• Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
• Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods

Related

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Dropout as a Structured Shrinkage Prior
• Dropout with Expectation-linear Regularization
• Generalized Dropout