Survey of Dropout Methods for Deep Neural Networks

2019/04/25 by Alex Labach, Hojjat Salehinejad, Labach, Alex +3 · 1 voice · 47 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Computer science #Convolutional neural network #Deep learning #Deep neural networks #Dropout (neural networks) #Gaussian Processes and Bayesian Inference #Inference #Machine learning #Regularization (linguistics) #cs.AI #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1904.13310

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2019/04/25 · arxiv created 2019/10/25 · arxiv updated 2020/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Dropout methods are a family of stochastic techniques used in neural network training or inference that have generated significant research interest and are widely used in practice. They have been successfully applied in neural network regularization, model compression, and in measuring the uncertainty of neural network outputs. While original formulated for dense neural network layers, recent advances have made dropout methods also applicable to convolutional and recurrent neural network layers. This paper summarizes the history of dropout methods, their various applications, and current areas of research interest. Important proposed methods are described in additional detail.

Citations

• Variational Dropout and the Local Reparameterization Trick
• ImageNet classification with deep convolutional neural networks

Cited by

• Improving action classification with brain-inspired deep networks
• Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks
• R-Drop: Regularized Dropout for Neural Networks
• Skyrmionic polarization textures in structured dielectric planar media
• Deep Neural Networks Inspired by Differential Equations
• RotationOut as a Regularization Method for Neural Network
• Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission
• FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
• Multitask LSTM for Arboviral Outbreak Prediction Using Public Health Data
• Locally Learned Synaptic Dropout for Complete Bayesian Inference
• EventDrop: data augmentation for event-based learning
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization
• EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks
• Hadamard product in deep learning: Introduction, Advances and Challenges

Discussions

• Survey of Dropout Methods for Deep Neural Networks [hn, 88 points, 5 comments]

Related

• Analytic theory of dropout regularization
• Implicit regularization of dropout
• Dropout with Tabu Strategy for Regularizing Deep Neural Networks
• Generalized Dropout
• Dropout in Training Neural Networks: Flatness of Solution and Noise Structure