Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD

2020/06/15 by Ruosi Wan, Zhanxing Zhu, Wan, Ruosi +5
Computer Science · Mathematics · Physics and Astronomy · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.08419

Theoretical analysis on joint effect of normalization and weight decay

openalex publication_date 2020/06/15 · openalex created_date 2020/10/08 · arxiv created 2020/11/27 · arxiv updated 2020/11/30 · openalex updated_date 2026/07/28

Abstract

In this work, we comprehensively reveal the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum), named as Spherical Motion Dynamics (SMD). Most related works study SMD by focusing on "effective learning rate" in "equilibrium" condition, where weight norm remains unchanged. However, their discussions on why equilibrium condition can be reached in SMD is either absent or less convincing. Our work investigates SMD by directly exploring the cause of equilibrium condition. Specifically, 1) we introduce the assumptions that can lead to equilibrium condition in SMD, and prove that weight norm can converge at linear rate with given assumptions; 2) we propose "angular update" as a substitute for effective learning rate to measure the evolving of neural network in SMD, and prove angular update can also converge to its theoretical value at linear rate; 3) we verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our theoretical findings agree well with empirical observations.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ImageNet Large Scale Visual Recognition Challenge
• Instance Normalization: The Missing Ingredient for Fast Stylization
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• L2 Regularization versus Batch and Weight Normalization
• Escaping Saddles with Stochastic Gradients
• How Does Batch Normalization Help Optimization?
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Angle-based Search Space Shrinking for Neural Architecture Search

Related

• Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
• Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
• Spherical Perspective on Learning with Normalization Layers
• Spherical Convolutional Neural Networks: Stability to Perturbations in SO(3)
• Idealized Baroclinic Wave Test Results