Generalized Gradient Norm Clipping & Non-Euclidean (L₀,L₁)-Smoothness

2025/06/02 by Thomas Pethick, Pethick, Thomas, Wanyun Xie +10 · 1 voice · 8 citations
Computer Science · Engineering · Mathematics · #Advanced Numerical Analysis Techniques #Clipping (morphology) #Connection (principal bundle) #Convergence (economics) #Gradient descent #Norm (philosophy) #Numerical methods in inverse problems #Optimization and Variational Analysis #Rate of convergence #Stochastic gradient descent

paper · pdf · doi:10.48550/arxiv.2506.01913

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of (L₀,L₁)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal O(n^-1/4) convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

Citations

• Scalable Optimization in the Modular Norm

Cited by

• Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
• Frank-Wolfe Algorithms for (L0, L1)-smooth functions
• Beyond the Ideal: Analyzing the Inexact Muon Update
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
• Adaptive Conditional Gradient Descent
• Optimal Scaling Needs Optimal Norm
• Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
• On the Interaction of Noise, Compression Role, and Adaptivity under (L₀, L₁)-Smoothness: An SDE-based Approach
• On the Role of Batch Size in Stochastic Conditional Gradient Methods

Discussions

• My paper on Generalized Gradient Norm Clipping & Non-Euclidean (L0, L1)-Smoothness (together with collaborators from EPFL) was accepted as an oral at NeurIPS! We extend the theory for our Scion algori [bsky, 16 points, 1 comments]

Related

• Variance-reduced Clipping for Non-convex Optimization
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• Methods for Convex (L₀,L₁)-Smooth Optimization: Clipping, Acceleration, and Adaptivity
• Robustness to Unbounded Smoothness of Generalized SignSGD
• Mirror Descent Under Generalized Smoothness