Practical Gauss-Newton Optimisation for Deep Learning

2017/06/12 by Aleksandar Botev, Hippolyt Ritter, Botev, Aleksandar +3 · 1 voice · 5 citations
Computer Science · Physics and Astronomy · #Blind Source Separation Techniques #Model Reduction and Neural Networks #Neural Networks and Applications #stat.ML

paper · pdf · doi:10.48550/arxiv.1706.03662

openalex publication_date 2017/06/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a labo- rious process, our approach can provide good performance even when used with default set- tings. A side result of our work is that for piecewise linear transfer functions, the net- work objective function can have no differ- entiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

Cited by

• CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regression
• Non-Asymptotic Optimization and Generalization Bounds for Stochastic Gauss-Newton in Overparameterized Models
• Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation
• Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
• Revisiting Data Attribution for Influence Functions

Discussions

• update: i have invented something which already exists arxiv.org/pdf/1706.03662 [bsky, 39 points, 4 comments]

Related

• Exact Gauss-Newton Optimization for Training Deep Neural Networks
• Towards a Mathematical Understanding of the Difficulty in Learning with Feedforward Neural Networks
• Newton Methods for Convolutional Neural Networks
• Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks
• Block-diagonal Hessian-free Optimization for Training Neural Networks