2023/09/21 by Philip M. Long, Peter L. Bartlett, Long, Philip M. +1 · 5 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2309.12488
openalex publication_date 2023/09/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size η, the operator norm of the Hessian of the loss grows until it approximately reaches 2/η, after which it fluctuates around this value. The quantity 2/η has been called the "edge of stability" based on consideration of a local quadratic approximation of the loss. We perform a similar calculation to arrive at an "edge of stability" for Sharpness-Aware Minimization (SAM), a variant of GD which has been shown to improve its generalization. Unlike the case for GD, the resulting SAM-edge depends on the norm of the gradient. Using three deep learning training tasks, we see empirically that SAM operates on the edge of stability identified by this analysis.