Muon is Scalable for LLM Training

2025/02/24 by Jingyuan Liu, Jianlin Su, Liu, Jingyuan +49 · 68 citations
Engineering · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Particle Detector Development and Performance #Particle accelerators and beam dynamics #Superconducting Materials and Applications

paper · pdf · doi:10.48550/arxiv.2502.16982

openalex publication_date 2025/02/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves ∼ 2× computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

Cited by

• Fast Organic Crystal Structure Prediction with Unit Cell Flow Matching
• Attention Residuals
• Scale Weight Decay and Train Better
• Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
• Autoregressive One-Step Generative Modeling for Dynamical System Forecasting
• Spherical Leech Quantization for Visual Tokenization and Generation
• Optimizing Rank for High-Fidelity Implicit Neural Representations
• Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
• FOAM: Blocked State Folding for Memory-Efficient LLM Training
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
• Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Register Any Point: Scaling 3D Point Cloud Registration by Flow Matching
• Dion2: A Simple Method to Shrink Matrix in Muon
• HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention
• On the Origin of Algorithmic Progress in AI
• ROOT: Robust Orthogonalized Optimizer for Neural Network Training
• A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descent
• Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning
• Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates
• MuonAll: Muon Variant for Efficient Finetuning of Large Language Models
• Next-Latent Prediction Transformers Learn Compact World Models
• Motif 2 12.7B technical report
• Federated Stochastic Minimax Optimization under Heavy-Tailed Noises
• Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
• Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing
• Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?
• FedMuon: Accelerating Federated Learning with Matrix Orthogonalization
• BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training
• Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence
• Beyond the Ideal: Analyzing the Inexact Muon Update
• Unbiased Gradient Low-Rank Projection
• MARS-M: When Variance Reduction Meets Matrices
• MuonBP: Faster Muon via Block-Periodic Orthogonalization
• Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Training
• Cautious Weight Decay
• An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
• POME: Post Optimization Model Edit via Muon-style Projection
• NorMuon: Making Muon more efficient and scalable
• Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization
• Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
• Optimal Scaling Needs Optimal Norm
• REG: A Regularization Optimizer for Robust Training Dynamics
• On Provable Benefits of Muon in Federated Learning
• Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization
• DeMuon: A Decentralized Muon for Matrix Optimization over Graphs
• FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
• Conda: Column-Normalized Adam for Training Large Language Models Faster
• PreScope: Unleashing the Power of Prefetching for Resource-Constrained MoE Inference
• Effective Quantization of Muon Optimizer States
• Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
• Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
• Linear Power System Modeling and Analysis Across Wide Operating Ranges: A Hierarchical Neural State-Space Equation Approach
• Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
• ENSAM: an efficient foundation model for interactive segmentation of 3D medical images
• On the Convergence of Muon and Beyond
• Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO
• Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
• The Optimiser Hidden in Plain Sight: Training with the Loss Landscape's Induced Metric
• Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
• AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates
• Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees
• SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
• Kimi K2: Open Agentic Intelligence
• Mellum2 Technical Report
• PRAGMA: Revolut Foundation Model

Related

• NorMuon: Making Muon more efficient and scalable
• SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
• Muon: Training and Trade-offs with Latent Attention and MoE
• Effective Quantization of Muon Optimizer States
• MuonAll: Muon Variant for Efficient Finetuning of Large Language Models