Volumization as a Natural Generalization of Weight Decay

2020/03/25 by Liu Ziyin, Zihao Wang, Ziyin, Liu +5
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2003.11243

openalex publication_date 2020/03/25 · openalex created_date 2020/04/03 · openalex updated_date 2026/07/28

Abstract

We propose a novel regularization method, called volumization, for neural networks. Inspired by physics, we define a physical volume for the weight parameters in neural networks, and we show that this method is an effective way of regularizing neural networks. Intuitively, this method interpolates between an L₂ and L_∞ regularization. Therefore, weight decay and weight clipping become special cases of the proposed algorithm. We prove, on a toy example, that the essence of this method is a regularization technique to control bias-variance tradeoff. The method is shown to do well in the categories where the standard weight decay method is shown to work well, including improving the generalization of networks and preventing memorization. Moreover, we show that the volumization might lead to a simple method for training a neural network whose weight is binary or ternary.

Citations

• Three unfinished works on the optimal storage capacity of networks
• Gradient-based learning applied to document recognition
• A Survey on Methods and Theories of Quantized Neural Networks
• Three Mechanisms of Weight Decay Regularization
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Learning Not to Learn in the Presence of Noisy Labels
• Learning Multiple Layers of Features from Tiny Images

Related

• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
• Three Mechanisms of Weight Decay Regularization
• Neural Network Regularization via Robust Weight Factorization
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Weight Conditioning for Smooth Optimization of Neural Networks