The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size

2018/11/16 by Vardan Papyan, Papyan, Vardan · 15 citations
Computer Science · Mathematics · Physics and Astronomy · #Generative Adversarial Networks and Image Synthesis #Model Reduction and Neural Networks #Neural Networks and Applications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1811.07062

arxiv created 2019/06/03 · arxiv updated 2019/06/04

Abstract

We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits "spiked" behavior, with several outliers isolated from a continuous bulk. We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.

Cited by

• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• The Geometry of Sign Gradient Descent
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• Analysis of stochastic Lanczos quadrature for spectrum approximation
• Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Does the Data Induce Capacity Control in Deep Learning?
• Beyond Random Matrix Theory for Deep Networks
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Communication Efficient, Differentially Private Distributed Optimization using Correlation-Aware Sketching
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Directional Pruning of Deep Neural Networks
• Flatness is a False Friend

Related

• Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods
• HessFormer: Hessians at Foundation Scale
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density