Inherent Noise in Gradient Based Methods

2020/05/26 by Arushi Gupta, Gupta, Arushi
Computer Science · Mathematics · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2005.12743

arxiv created 2020/05/26 · openalex publication_date 2020/05/26 · arxiv updated 2020/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Previous work has examined the ability of larger capacity neural networks to generalize better than smaller ones, even without explicit regularizers, by analyzing gradient based algorithms such as GD and SGD. The presence of noise and its effect on robustness to parameter perturbations has been linked to generalization. We examine a property of GD and SGD, namely that instead of iterating through all scalar weights in the network and updating them one by one, GD (and SGD) updates all the parameters at the same time. As a result, each parameter wⁱ calculates its partial derivative at the stale parameter w_t, but then suffers loss L(\mathbfw_t+1). We show that this causes noise to be introduced into the optimization. We find that this noise penalizes models that are sensitive to perturbations in the weights. We find that penalties are most pronounced for batches that are currently being used to update, and are higher for larger models.

Citations

• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Dropout Training as Adaptive Regularization
• Sharp Minima Can Generalize For Deep Nets
• Three Factors Influencing Minima in SGD
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Understanding deep learning requires rethinking generalization
• Weight Uncertainty in Neural Networks

Related

• On Linear Stability of SGD and Input-Smoothness of Neural Networks
• On the Noisy Gradient Descent that Generalizes as SGD
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Explicit Regularization in Overparametrized Models via Noise Injection
• Noisy Truncated SGD: Optimization and Generalization