Towards Quantifying the Hessian Structure of Neural Networks

2025/05/05 by Dong, Zhaorui, Zhang, Yushun, Yao, Jianfeng +1 · 2 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2505.02809

Abstract

Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture of two forces: a ``static force'' rooted in the architecture design, and a ''dynamic force'' arisen from training. We then provide a rigorous theoretical analysis of ''static force'' at random initialization. We study linear models and 1-hidden-layer networks for classification tasks with C classes. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises as C becomes large. Our findings reveal that C is one primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a large C exceeding 10⁴.

Cited by

• Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
• On the Convergence Analysis of Muon

Related

• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• Hessian Eigenvectors and Principal Component Analysis of Neural Network Weight Matrices
• Local properties of neural networks through the lens of layer-wise Hessians
• The Hessian perspective into the Nature of Convolutional Neural Networks