Towards Quantifying the Hessian Structure of Neural Networks

2025/05/05 by Zhaorui Dong, Yushun Zhang, Dong, Zhaorui +5 · 2 citations
Computer Science · #Artificial neural network #Diagonal #FOS: Computer and information sciences #FOS: Mathematics #Feature (linguistics) #Hessian equation #Hessian matrix #Limit (mathematics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Matrix (chemical analysis) #Neural Networks and Applications #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2505.02809

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture of two forces: a ``static force'' rooted in the architecture design, and a ''dynamic force'' arisen from training. We then provide a rigorous theoretical analysis of ''static force'' at random initialization. We study linear models and 1-hidden-layer networks for classification tasks with C classes. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises as C becomes large. Our findings reveal that C is one primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a large C exceeding 10⁴.

Citations

• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
• Kimi K2: Open Agentic Intelligence
• ASGO: Adaptive Structured Gradient Optimization
• The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
• Muon is Scalable for LLM Training
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
• Understanding Adam Requires Better Rotation Dependent Assumptions
• What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
• SOAP: Improving and Stabilizing Shampoo using Adam
• Adam-mini: Use Fewer Learning Rates To Gain More
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
• Why Transformers Need Adam: A Hessian Perspective
• Towards Quantifying the Preconditioning Effect of Adam
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Optimal Diagonal Preconditioning
• Adaptive Gradient Methods at the Edge of Stability
• Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability
• Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction
• Understanding Gradient Descent on Edge of Stability in Deep Learning
• How Do Vision Transformers Work?
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Gradient Descent on Neural Networks Typically Occurs at the Edge of\n Stability
• A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization
• On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Practical Quasi-Newton Methods for Training Deep Neural Networks
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Optimization for deep learning: theory and algorithms
• Fantastic Generalization Measures and Where to Find Them
• Emergent properties of the local geometry of neural loss landscapes
• How noise affects the Hessian spectrum in overparameterized neural networks
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Modular Block-diagonal Curvature Approximations for Feedforward Architectures
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Gradient Descent Happens in a Tiny Subspace
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Negative eigenvalues of the Hessian in deep neural networks
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Natural Neural Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Asymptotic Spectra of Matrix-Valued Functions of Independent Random Matrices and Free Probability
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• On the Distribution of the Roots of Certain Symmetric Matrices

Cited by

• Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
• On the Convergence Analysis of Muon

Related

• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• Hessian Eigenvectors and Principal Component Analysis of Neural Network Weight Matrices
• Local properties of neural networks through the lens of layer-wise Hessians
• The Hessian perspective into the Nature of Convolutional Neural Networks