The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD\n Training and Sample Size

2018/11/16 by Vardan Papyan, Papyan, Vardan · 13 citations
Physics and Astronomy · Computer Science · #Model Reduction and Neural Networks #Generative Adversarial Networks and Image Synthesis #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1811.07062

Abstract

We apply state-of-the-art tools in modern high-dimensional numerical linear\nalgebra to approximate efficiently the spectrum of the Hessian of modern\ndeepnets, with tens of millions of parameters, trained on real data. Our\nresults corroborate previous findings, based on small-scale networks, that the\nHessian exhibits "spiked" behavior, with several outliers isolated from a\ncontinuous bulk. We decompose the Hessian into different components and study\nthe dynamics with training and sample size of each term individually.\n

Cited by

• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• The Geometry of Sign Gradient Descent
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• Analysis of stochastic Lanczos quadrature for spectrum approximation
• Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Does the Data Induce Capacity Control in Deep Learning?
• Beyond Random Matrix Theory for Deep Networks
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank\n structure
• Communication Efficient, Differentially Private Distributed Optimization using Correlation-Aware Sketching
• Exploring Weight Importance and Hessian Bias in Model Pruning

Related

• Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods
• HessFormer: Hessians at Foundation Scale
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• An Investigation into Neural Net Optimization via Hessian Eigenvalue\n Density