Gathering and Exploiting Higher-Order Information when Training Large Structured Models

2023/12/06 by Pierre Wolinski, Wolinski, Pierre · 1 voice
Computer Science · #Machine Learning and Algorithms #Machine Learning and Data Classification #Neural Networks and Applications

paper · doi:10.1007/978-3-032-06078-5_25

openalex publication_date 2025/09/29 · crossref created 2025/09/29 · crossref deposited 2025/09/29 · crossref issued 2025/09/30 · crossref published 2025/09/30 · crossref published-online 2025/09/30 · crossref indexed 2025/09/30 · openalex created_date 2025/10/10 · crossref published-print 2026/01/01 · openalex updated_date 2026/07/29

Abstract

When training large models, such as neural networks, the full derivatives of order 2 and beyond are usually inaccessible, due to their computational cost. Therefore, among the second-order optimization methods, it is common to bypass the computation of the Hessian by using first-order information, such as the gradient of the parameters (e.g., quasi-Newton methods) or the activations (e.g., K-FAC). In this paper, we focus on the exact and explicit computation of projections of the Hessian and higher-order derivatives on well-chosen subspaces relevant for optimization. Namely, for a given partition of the set of parameters, we compute tensors that can be seen as "higher-order derivatives according to the partition", at a reasonable cost as long as the number of subsets of the partition remains small. Then, we give some examples of how these tensors can be used. First, we show how to compute a learning rate per subset of parameters, which can be used for hyperparameter tuning. Second, we show how to use these tensors at order 2 to construct an optimization method that uses information contained in the Hessian. Third, we show how to use these tensors at order 3 (information contained in the third derivative of the loss) to regularize this optimization method. The resulting training step has several interesting properties, including: it takes into account long-range interactions between the layers of the trained neural network, which is usually not the case in similar methods (e.g., K-FAC); the trajectory of the optimization is invariant under affine layer-wise reparameterization.

Citations

• Natural Gradient Works Efficiently in Learning
• Gradient-based learning applied to document recognition
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• Tensor Normal Training for Deep Learning Models
• Sketched Newton-Raphson
• Practical Quasi-Newton Methods for Training Deep Neural Networks
• Stochastic Subspace Cubic Newton Method
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• RSN: Randomized Subspace Newton
• Similarity of Neural Network Representations Revisited
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Are All Layers Created Equal?
• Fine-Grained Analysis of Optimization and Generalization for\n Overparameterized Two-Layer Neural Networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Block Mean Approximation for Efficient Second Order Optimization
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Riemannian metrics for neural networks I: feedforward networks

Discussions

• 📖 New LAMSADE paper: Gathering and Exploiting Higher-Order Information when Training Large Structured Models. (ECML/PKDD (4)) https://doi.org/10.1007/978-3-032-06078-5_25 [bsky, 0 points, 0 comments]