Muon is Scalable for LLM Training

2025/02/24 by Jingyuan Liu, Liu, Jingyuan, Jianlin Su +57 · 1 voice · 110 citations
Computer Science · Engineering · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Particle Detector Development and Performance #Particle accelerators and beam dynamics #Superconducting Materials and Applications #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.16982

openalex publication_date 2025/02/24 · arxiv published 2025/02/24 · arxiv updated 2025/02/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves ∼ 2× computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

Cited by

• Fast Organic Crystal Structure Prediction with Unit Cell Flow Matching
• Attention Residuals
• Scale Weight Decay and Train Better
• Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
• Autoregressive One-Step Generative Modeling for Dynamical System Forecasting
• Spherical Leech Quantization for Visual Tokenization and Generation
• Optimizing Rank for High-Fidelity Implicit Neural Representations
• Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
• FOAM: Blocked State Folding for Memory-Efficient LLM Training
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
• Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Register Any Point: Scaling 3D Point Cloud Registration by Flow Matching
• Dion2: A Simple Method to Shrink Matrix in Muon
• HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention
• On the Origin of Algorithmic Progress in AI
• ROOT: Robust Orthogonalized Optimizer for Neural Network Training
• A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descent
• Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning
• Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates
• MuonAll: Muon Variant for Efficient Finetuning of Large Language Models
• Next-Latent Prediction Transformers Learn Compact World Models
• Motif 2 12.7B technical report
• Federated Stochastic Minimax Optimization under Heavy-Tailed Noises
• Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
• Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing
• Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?
• FedMuon: Accelerating Federated Learning with Matrix Orthogonalization
• BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training
• Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence
• Beyond the Ideal: Analyzing the Inexact Muon Update
• Unbiased Gradient Low-Rank Projection
• MARS-M: When Variance Reduction Meets Matrices
• MuonBP: Faster Muon via Block-Periodic Orthogonalization
• Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Training
• Cautious Weight Decay
• An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
• POME: Post Optimization Model Edit via Muon-style Projection
• NorMuon: Making Muon more efficient and scalable
• Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization
• Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
• Optimal Scaling Needs Optimal Norm
• REG: A Regularization Optimizer for Robust Training Dynamics
• On Provable Benefits of Muon in Federated Learning
• Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization
• DeMuon: A Decentralized Muon for Matrix Optimization over Graphs
• FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
• Conda: Column-Normalized Adam for Training Large Language Models Faster
• PreScope: Unleashing the Power of Prefetching for Resource-Constrained MoE Inference
• Effective Quantization of Muon Optimizer States
• Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
• Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
• Linear Power System Modeling and Analysis Across Wide Operating Ranges: A Hierarchical Neural State-Space Equation Approach
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• ENSAM: an efficient foundation model for interactive segmentation of 3D medical images
• On the Convergence of Muon and Beyond
• Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO
• Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
• The Optimiser Hidden in Plain Sight: Training with the Loss Landscape's Induced Metric
• Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
• AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
• Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees
• SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
• Kimi K2: Open Agentic Intelligence
• Mellum2 Technical Report
• PRAGMA: Revolut Foundation Model
• AdaMuon: Adaptive Muon Optimizer
• Low-rank Momentum Factorization for Memory Efficient Training
• GradMetaNet: An Equivariant Architecture for Learning on Gradients
• SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement
• AuON: A Linear-time Alternative to Semi-Orthogonal Momentum Updates
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
• The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
• Reparameterized LLM Training via Orthogonal Equivalence Transformation
• Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
• Muon Optimizes Under Spectral Norm Constraints
• Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
• Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials
• FlashMoE: Fast Distributed MoE in a Single Kernel
• Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
• Taming LLMs by Scaling Learning Rates with Gradient Grouping
• GradPower: Powering Gradients for Faster Language Model Pre-Training
• MuLoCo: Muon is a practical inner optimizer for DiLoCo
• On the Convergence Analysis of Muon
• Towards Quantifying the Hessian Structure of Neural Networks
• In Search of Adam's Secret Sauce
• A Continuous-Time Analysis of Smoothed Matrix-Polar Spectral Gradient Flows for Muon-Type Optimization
• The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
• CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalization
• Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)
• Reassessing Muon for Matrix Factorization
• Practical Efficiency of Muon for Pretraining
• Scalable Physics-Inspired Transformers for Spin Glasses
• OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning
• Celo2: Towards Learned Optimization Free Lunch
• The Newton-Muon Optimizer
• Spectral Scaling Laws of Muon
• Training-Free Looped Transformers
• Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
• Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima
• TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
• LoopMTP: A looped transformer guided by latent multi-token prediction
• Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon
• Kimi-Audio Technical Report
• Dion: Distributed Orthonormalized Updates
• Muon Optimizer Accelerates Grokking
• Kimi-VL Technical Report

Discussions

• Muon is Scalable for LLM Training arxiv.org/abs/2502.16982 Project page: github.com/MoonshotAI/M... Checkpoints are available on Huggingface: huggingface.co/moonshotai/M... [bsky, 1 points, 0 comments]

Related

• NorMuon: Making Muon more efficient and scalable
• SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
• Muon: Training and Trade-offs with Latent Attention and MoE
• Effective Quantization of Muon Optimizer States
• MuonAll: Muon Variant for Efficient Finetuning of Large Language Models