Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond

2023/05/22 by Itai Kreisler, Kreisler, Itai, Mor Shpigel Nacson +5 · 5 citations
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2305.13064

openalex publication_date 2023/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent research shows that when Gradient Descent (GD) is applied to neural networks, the loss almost never decreases monotonically. Instead, the loss oscillates as gradient descent converges to its ''Edge of Stability'' (EoS). Here, we find a quantity that does decrease monotonically throughout GD training: the sharpness attained by the gradient flow solution (GFS)-the solution that would be obtained if, from now until convergence, we train with an infinitesimal step size. Theoretically, we analyze scalar neural networks with the squared loss, perhaps the simplest setting where the EoS phenomena still occur. In this model, we prove that the GFS sharpness decreases monotonically. Using this result, we characterize settings where GD provably converges to the EoS in scalar networks. Empirically, we show that GD monotonically decreases the GFS sharpness in a squared regression model as well as practical neural network architectures.

Cited by

• Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region
• Variational Learning Finds Flatter Solutions at the Edge of Stability
• Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More
• SGD at the Edge of Stability: The Stochastic Sharpness Gap
• Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes

Related

• Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Sharpness-Aware Minimization and the Edge of Stability
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes