A Bayesian encourages dropout

2014/12/22 by Shin‐ichi Maeda, Shin-ichi Maeda, Maeda, Shin-ichi · 35 citations
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Bayesian probability #Computer science #Dropout (neural networks) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine learning #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Overfitting #Regularization (linguistics) #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1412.7003

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2014/12/22 · arxiv created 2014/12/30 · arxiv updated 2014/12/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of weight parameters and prediction after learning. The experiment result also encourages the optimization of the dropout.

Citations

• Spatially-sparse convolutional neural networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Dropout Training as Adaptive Regularization

Cited by

• BayesGrad: Explaining Predictions of Graph Convolutional Networks
• Regularization for Deep Learning: A Taxonomy
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization
• CODA: Constructivism Learning for Instance-Dependent Dropout Architecture Construction

Related

• Dropout as a Bayesian Approximation: Appendix
• Pushing the bounds of dropout
• Dropout Inference with Non-Uniform Weight Scaling
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Machine Learning's Dropout Training is Distributionally Robust Optimal