Structured Sparsity Inducing Adaptive Optimizers for Deep Learning

2021/02/07 by Tristan Deleu, Yoshua Bengio, Deleu, Tristan +1 · 2 citations
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning and ELM #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2102.03869

openalex publication_date 2021/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The parameters of a neural network are naturally organized in groups, some of which might not contribute to its overall performance. To prune out unimportant groups of parameters, we can include some non-differentiable penalty to the objective function, and minimize it using proximal gradient methods. In this paper, we derive the weighted proximal operator, which is a necessary component of these proximal methods, of two structured sparsity inducing penalties. Moreover, they can be approximated efficiently with a numerical solver, and despite this approximation, we prove that existing convergence guarantees are preserved when these operators are integrated as part of a generic adaptive proximal method. Finally, we show that this adaptive method, together with the weighted proximal operators derived here, is indeed capable of finding solutions with structure in their sparsity patterns, on representative examples from computer vision and natural language processing.

Citations

• Nearly unbiased variable selection under minimax concave penalty
• Monotone Operators and the Proximal Point Algorithm
• ImageNet classification with deep convolutional neural networks
• Non-asymptotic Analysis of Stochastic Methods for Non-Smooth Non-Convex\n Regularized Problems
• Learning Neural Causal Models from Unknown Interventions
• Proximal Adam: Robust Adaptive Update Scheme for Constrained Optimization
• A General Family of Stochastic Proximal Gradient Methods for Deep Learning

Cited by

• Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization
• A DC-Reformulation for Gradient-L⁰-Constrained Problems

Related

• Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization
• Embedding Differentiable Sparsity into Deep Neural Network
• Differentiable Sparsification for Deep Neural Networks
• Smoothing Proximal Gradient Method for General Structured Sparse Learning
• Minimum Variance Unbiased N:M Sparsity for the Neural Gradients