Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks

2025/01/15 by Pierfrancesco Beneventano, Beneventano, Pierfrancesco, Blake Woodworth +1 · 1 citation
Computer Science · #Advanced Mathematical Modeling in Engineering #Complexity and Algorithms in Graphs #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mobile Ad Hoc Networks #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2501.09137

openalex publication_date 2025/01/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the gradient descent (GD) dynamics of a depth-2 linear neural network with a single input and output. We show that GD converges at an explicit linear rate to a global minimum of the training loss, even with a large stepsize -- about 2/\textrmsharpness. It still converges for even larger stepsizes, but may do so very slowly. We also characterize the solution to which GD converges, which has lower norm and sharpness than the gradient flow solution. Our analysis reveals a trade off between the speed of convergence and the magnitude of implicit regularization. This sheds light on the benefits of training at the ``Edge of Stability'', which induces additional regularization by delaying convergence and may have implications for training more complex models.

Cited by

• Lyapunov stability of the Euler method

Related

• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks
• Convergence of gradient descent for learning linear neural networks
• Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond
• On the Convergence of Gradient Descent for Large Learning Rates