On the Convergence of Muon and Beyond

2025/09/19 by Da Chang, Yongxiang Liu, Chang, Da +3 · 5 citations
Physics and Astronomy · Engineering · Computer Science · #Particle physics theoretical and experimental studies #Muon and positron interactions and applications #Computational Physics and Python Applications

paper · pdf · doi:10.48550/arxiv.2509.15816

Abstract

The Muon optimizer has demonstrated remarkable empirical success in handling matrix-structured parameters for training neural networks. However, a significant gap remains between its practical performance and theoretical understanding. Existing analyses show that the Muon variants achieve only a suboptimal ergodic convergence rate of O(T^-1/4) in stochastic non-convex settings, where T denotes the number of iterations. To study the theoretical limits of Muon, we analyze two momentum-based variance-reduced variants: the one-batch Muon-MVR1 and the two-batch Muon-MVR2. We provide the first rigorous proof that, under horizon-free learning-rate schedules, variance reduction enables Muon-MVR2 to attain the optimal anytime convergence rate \widetildeO(T^-1/3), matching the lower bound for this problem class. Under the Polyak--Łojasiewicz (PL) condition, we establish anytime guarantees for Muon-MVR1 and Muon-MVR2: they attain best-iterate rates of \widetildeO(T^-1/4) and \widetildeO(T^-1/3) for the expected square-root suboptimality, and, given an additional uniform gradient bound along the iterates, achieve last-iterate rates of O(T^-1/4) and O(T^-1/3) for the objective gap, respectively. Experiments on CIFAR-10 and C4 support the practical effectiveness of the proposed variance-reduced Muon variants. Code is available at \hrefhttps://github.com/MaeChd/MUON-MVRMuon-MVR Codebase.

Citations

• AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
• AdaMuon: Adaptive Muon Optimizer
• AuON: A Linear-time Alternative to Semi-Orthogonal Momentum Updates
• Muon Optimizes Under Spectral Norm Constraints
• Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
• PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
• Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
• Practical Efficiency of Muon for Pretraining
• ASGO: Adaptive Structured Gradient Optimization
• Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization
• Muon is Scalable for LLM Training
• COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs
• Training Deep Learning Models with Norm-Constrained LMOs
• Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
• A Note on the Convergence of Muon
• Cautious Optimizers: Improving Training with One Line of Code
• MARS: Unleashing the Power of Variance Reduction for Training Large Models
• Old Optimizer, New Norm: An Anthology
• Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods
• Convergence of Adam Under Relaxed Assumptions
• Symbolic Discovery of Optimization Algorithms
• Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization
• Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
• Training Compute-Optimal Large Language Models
• A Novel Convergence Analysis for Algorithms of the Adam Family
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
• SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients
• Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum
• Momentum Improves Normalized SGD
• Lower Bounds for Non-Convex Stochastic Optimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• On the Variance of the Adaptive Learning Rate and Beyond
• Momentum-Based Variance Reduction in Non-Convex SGD
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Stochastic Nested Variance Reduction for Nonconvex Optimization
• Shampoo: Preconditioned Stochastic Tensor Optimization
• On the Convergence of Adam and Beyond
• Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition
• Deep Residual Learning for Image Recognition
• Proximal alternating linearized minimization for nonconvex and nonsmooth problems

Cited by

• Scale Weight Decay and Train Better
• Muon is Provably Faster with Momentum Variance Reduction
• Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval
• Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data

Related

• On the Convergence Analysis of Muon
• AuON: A Linear-time Alternative to Semi-Orthogonal Momentum Updates
• Muon is Provably Faster with Momentum Variance Reduction
• Beyond the Ideal: Analyzing the Inexact Muon Update
• LiMuon: Light and Fast Muon Optimizer for Large Models