Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization

2020/10/31 by Jiyang Xie, Zhanyu Ma, Jianjun Lei +7 · 1 citation
Computer Science · Mathematics · #Artificial intelligence #Bayesian optimization #Bayesian probability #Computer science #Data Stream Mining Techniques #Dropout (neural networks) #Machine Learning and Algorithms #Machine Learning and Data Classification #Machine learning #cs.LG #stat.ML

paper · pdf · doi:10.1109/tpami.2021.3083089

Accepted by IEEE TPAMI, 2021

openalex publication_date 2021/01/01 · arxiv created 2021/08/10 · arxiv updated 2021/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance of DNNs. The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. Specifically, the distribution parameters are optimized by stochastic gradient variational Bayes in order to carry out an end-to-end training. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets (five small-scale datasets and two large-scale datasets) with various base models. The advanced dropout outperforms all the referred techniques on all the datasets. We further compare the effectiveness ratios and find that advanced dropout achieves the highest one on most cases. Next, we conduct a set of analysis of dropout rate characteristics, including convergence of the adaptive dropout rate, the learned distributions of dropout masks, and a comparison with dropout rate generation without an explicit distribution. In addition, the ability of overfitting prevention is evaluated and confirmed. Finally, we extend the application of the advanced dropout to uncertainty inference, network pruning, text classification, and regression. The proposed advanced dropout is also superior to the corresponding referred methods. Codes are available at https://github.com/PRIS-CV/AdvancedDropout.

Citations

• A Bayesian encourages dropout
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Variational Dropout and the Local Reparameterization Trick
• Improving neural networks by preventing co-adaptation of feature detectors
• Auto-Encoding Variational Bayes
• ImageNet Large Scale Visual Recognition Challenge
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Predictive Uncertainty Estimation via Prior Networks
• Survey of Dropout Methods for Deep Neural Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Last-layer committee machines for uncertainty estimations of benthic imagery