Nonconvex regularization for sparse neural networks

2020/04/24 by Konstantin Pieper, Pieper, Konstantin, Armenak Petrosyan +1 · 1 citation
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Photoacoustic and Ultrasonic Imaging #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2004.11515

openalex publication_date 2020/04/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Convex ℓ₁ regularization using an infinite dictionary of neurons has been suggested for constructing neural networks with desired approximation guarantees, but can be affected by an arbitrary amount of over-parametrization. This can lead to a loss of sparsity and result in networks with too many active neurons for the given data, in particular if the number of data samples is large. As a remedy, in this paper, a nonconvex regularization method is investigated in the context of shallow ReLU networks: We prove that in contrast to the convex approach, any resulting (locally optimal) network is finite even in the presence of infinite data (i.e., if the data distribution is known and the limiting case of infinite samples is considered). Moreover, we show that approximation guarantees and existing bounds on the network size for finite data are maintained.

Cited by

• Solving Nonlinear PDEs with Sparse Radial Basis Function Networks

Related

• On Sparsity in Overparametrised Shallow ReLU Networks
• Transformed ℓ₁ Regularization for Learning Sparse Deep Neural Networks
• Global Minimizers of ℓ^p-Regularized Objectives Yield the Sparsest ReLU Neural Networks
• Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
• Path Regularization: A Convexity and Sparsity Inducing Regularization\n for Parallel ReLU Networks