Convergence of optimizers implies eigenvalues filtering at equilibrium

2025/10/10 by Jerome Bolte, Quoc-Tung Le, Bolte, Jerome +3 · 1 voice
Computer Science · Mathematics · #cs.LG #math.DS #math.OC

paper · pdf · doi:10.48550/arxiv.2510.09034

Abstract

Ample empirical evidence in deep neural network training suggests that a variety of optimizers tend to find nearly global optima. In this article, we adopt the reversed perspective that convergence to an arbitrary point is assumed rather than proven, focusing on the consequences of this assumption. From this viewpoint, in line with recent advances on the edge-of-stability phenomenon, we argue that different optimizers effectively act as eigenvalue filters determined by their hyperparameters. Specifically, the standard gradient descent method inherently avoids the sharpest minima, whereas Sharpness-Aware Minimization (SAM) algorithms go even further by actively favoring wider basins. Inspired by these insights, we propose two novel algorithms that exhibit enhanced eigenvalue filtering, effectively promoting wider minima. Our theoretical analysis leverages a generalized Hadamard--Perron stable manifold theorem and applies to general semialgebraic C² functions, without requiring additional non-degeneracy conditions or global Lipschitz bound assumptions. We support our conclusions with numerical experiments on feed-forward neural networks.

Citations

• Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
• Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD
• Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training
• Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
• Deep linear networks for regression are implicitly regularized towards flat minima
• High dimensional analysis reveals conservative sharpening and a stochastic edge of stability
• Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency
• Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy
• The Crucial Role of Normalization in Sharpness-Aware Minimization
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability
• SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• A Modern Look at the Relationship between Sharpness and Generalization
• Stability Analysis of Sharpness-Aware Minimization
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
• Adaptive Gradient Methods at the Edge of Stability
• On the Maximum Hessian Eigenvalue and Generalization
• Towards Understanding Sharpness-Aware Minimization
• Understanding Gradient Descent on Edge of Stability in Deep Learning
• Understanding the unstable convergence of gradient descent
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Heavy-ball Algorithms Always Escape Saddle Points
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Wide Residual Networks
• Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
• Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods
• Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka-Lojasiewicz inequality
• Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Łojasiewicz Inequality
• Geometric categories and o-minimal structures

Cited by

• Reachability of gradient dynamics

Discussions

• Straight to the top of the "to read" list: arxiv.org/pdf/2510.09034 [bsky, 6 points, 2 comments]

Related

• Sharpness-Aware Minimization and the Edge of Stability
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Understanding Edge-of-Stability Training Dynamics with a Minimalist Example
• Optimizers Qualitatively Alter Solutions And We Should Leverage This
• SAM operates far from home: eigenvalue regularization as a dynamical phenomenon