A Bayesian encourages dropout

2014/12/22 by Shin‐ichi Maeda, Maeda, Shin-ichi · 1 citation
Computer Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1412.7003

openalex publication_date 2014/12/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of weight parameters and prediction after learning. The experiment result also encourages the optimization of the dropout.

Citations

• Spatially-sparse convolutional neural networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Dropout Training as Adaptive Regularization

Cited by

• BayesGrad: Explaining Predictions of Graph Convolutional Networks

Related

• Dropout as a Bayesian Approximation: Appendix
• Pushing the bounds of dropout
• Dropout Inference with Non-Uniform Weight Scaling
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Machine Learning's Dropout Training is Distributionally Robust Optimal