Generalized Gradient Norm Clipping & Non-Euclidean (L₀,L₁)-Smoothness

2025/06/02 by Thomas Pethick, Pethick, Thomas, Wanyun Xie +10 · 1 voice · 7 citations
Mathematics · Engineering · Computer Science · #Numerical methods in inverse problems #Advanced Numerical Analysis Techniques #Optimization and Variational Analysis

paper · pdf · doi:10.48550/arxiv.2506.01913

Abstract

This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of (L₀,L₁)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal O(n^-1/4) convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

Citations

• Scalable Optimization in the Modular Norm

Cited by

• Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
• Frank-Wolfe Algorithms for (L0, L1)-smooth functions
• Beyond the Ideal: Analyzing the Inexact Muon Update
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
• Adaptive Conditional Gradient Descent
• Optimal Scaling Needs Optimal Norm
• Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives

Discussions

• My paper on Generalized Gradient Norm Clipping & Non-Euclidean (L0, L1)-Smoothness (together with collaborators from EPFL) was accepted as an oral at NeurIPS! We extend the theory for our Scion algori [bsky, 16 points, 1 comments]

Related

• Variance-reduced Clipping for Non-convex Optimization
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• Methods for Convex (L₀,L₁)-Smooth Optimization: Clipping, Acceleration, and Adaptivity
• Robustness to Unbounded Smoothness of Generalized SignSGD
• Mirror Descent Under Generalized Smoothness