Dropout: Explicit Forms and Capacity Control

2020/03/06 by Raman Arora, Arora, Raman, Peter L. Bartlett +5 · 8 citations
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2003.03397

openalex publication_date 2020/03/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.

Citations

• Learning with the Weighted Trace-norm under Arbitrary Sampling Distributions
• Learning Multiple Layers of Features from Tiny Images

Cited by

• The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Control
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime
• Overlap-Adaptive Regularization for Conditional Average Treatment Effect Estimation
• The Implicit and Explicit Regularization Effects of Dropout
• What training reveals about neural network complexity
• Implicit Regularization and Convergence for Weight Normalization
• How Does Mixup Help With Robustness and Generalization?
• The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and\n Regularization
• Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines

Related

• Implicit regularization of dropout
• Dropout with Expectation-linear Regularization
• Investigating the Relationship Between Dropout Regularization and Model Complexity in Neural Networks
• Surprising properties of dropout in deep networks
• Revisiting Structured Dropout