HessFormer: Hessians at Foundation Scale

2025/05/16 by Diego Granziol, Granziol, Diego · 1 citation
Computer Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.11564

openalex publication_date 2025/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the hundreds of billions of parameters, methods that rely on Hessian vector products, are still limited to run on a single GPU and thus cannot even work for models in the billion parameter range. We release a software package HessFormer, which integrates nicely with the well known Transformers package and allows for distributed hessian vector computation across a single node with multiple GPUs. Underpinning our implementation is a distributed stochastic lanczos quadrature algorithm, which we release for public consumption. Using this package we investigate the Hessian spectral density of the recent Deepseek 70bn parameter model.

Cited by

• Selecting Auxiliary Data via Neural Tangent Kernels for Low-Resource Domains

Related

• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD\n Training and Sample Size
• Shoot from the HIP: Hessian Interatomic Potentials without derivatives
• Efficient Computation of Hessian Matrices in TensorFlow
• Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source)
• Accelerating SGD for Distributed Deep-Learning Using Approximated Hessian Matrix