2016/09/26 by Xuezhe Ma, Yingkai Gao, Ma, Xuezhe +9 · 13 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Algorithm #Artificial intelligence #Artificial neural network #Benchmark (surveying) #Computer science #Domain Adaptation and Few-Shot Learning #Dropout (neural networks) #FOS: Computer and information sciences #Inference #Latent variable #Linearization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematical optimization #Mathematics #Nonlinear system #Regularization (linguistics) #Simple (philosophy) #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1609.08017
published in arXiv (Cornell University) (Cornell University) · Published as a conference paper at ICLR 2017. Camera-ready Version. 23 pages (paper + appendix)
openalex publication_date 2016/09/26 · arxiv created 2017/02/15 · arxiv updated 2017/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout's training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an explicit control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.