How noise affects the Hessian spectrum in overparameterized neural networks

2019/10/01 by Mingwei Wei, David J. Schwab, Wei, Mingwei +1 · 2 citations
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1910.00195

openalex publication_date 2019/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks with a degenerate valley in their loss landscape, SGD on average decreases the trace of the Hessian of the loss. We also generalize this result to other noise structures and show that isotropic noise in the non-degenerate subspace of the Hessian decreases its determinant. In addition to explaining SGDs role in sculpting the Hessian spectrum, this opens the door to new optimization approaches that may confer better generalization performance. We test our results with experiments on toy models and deep neural networks.

Citations

• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Three Factors Influencing Minima in SGD
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Gradient Descent Happens in a Tiny Subspace
• An Investigation into Neural Net Optimization via Hessian Eigenvalue\n Density
• Flat Minima

Cited by

• Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks
• Why flatness does and does not correlate with generalization for deep neural networks

Related

• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization
• Deep learning, stochastic gradient descent and diffusion maps
• SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects