The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and\n Regularization

2021/06/14 by Daniel LeJeune, Hamid Javadi, LeJeune, Daniel +3
Computer Science · Engineering · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2106.07769

openalex publication_date 2021/06/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Among the most successful methods for sparsifying deep (neural) networks are\nthose that adaptively mask the network weights throughout training. By\nexamining this masking, or dropout, in the linear case, we uncover a duality\nbetween such adaptive methods and regularization through the so-called\n"\η-trick" that casts both as iteratively reweighted optimizations. We show\nthat any dropout strategy that adapts to the weights in a monotonic way\ncorresponds to an effective subquadratic regularization penalty, and therefore\nleads to sparse solutions. We obtain the effective penalties for several\npopular sparsification strategies, which are remarkably similar to classical\npenalties commonly used in sparse optimization. Considering variational dropout\nas a case study, we demonstrate similar empirical behavior between the adaptive\ndropout method and classical methods on the task of deep network\nsparsification, validating our theory.\n

Citations

• Structured Sparse Principal Component Analysis
• Variational Dropout and the Local Reparameterization Trick
• Nearly unbiased variable selection under minimax concave penalty
• Robust Estimation of a Location Parameter
• Iteratively reweighted least squares minimization for sparse recovery
• Dropout Training as Adaptive Regularization
• Variational Dropout Sparsifies Deep Neural Networks
• The State of Sparsity in Deep Neural Networks
• Iterative hard thresholding for compressed sensing
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Dropout: Explicit Forms and Capacity Control

Related

• Surprising properties of dropout in deep networks
• On the Regularization Properties of Structured Dropout
• Learning Sparse Networks Using Targeted Dropout
• Decoupled Weight Decay for Any p Norm
• Implicit regularization of dropout