Decoupled Weight Decay for Any p Norm

2024/04/16 by Nadav Joseph Outmezguine, Noam Levi, Outmezguine, Nadav Joseph +1
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2404.10824

openalex publication_date 2024/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the success of deep neural networks (NNs) in a variety of domains, the computational and storage requirements for training and deploying large NNs have become a bottleneck for further improvements. Sparsification has consequently emerged as a leading approach to tackle these issues. In this work, we consider a simple yet effective approach to sparsification, based on the Bridge, or L_p regularization during training. We introduce a novel weight decay scheme, which generalizes the standard L₂ weight decay to any p norm. We show that this scheme is compatible with adaptive optimizers, and avoids the gradient divergence associated with 0

Related

• Understanding Decoupled and Early Weight Decay
• Adaptive Weight Decay for Deep Neural Networks
• Three Mechanisms of Weight Decay Regularization
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures