Effective Quantization of Muon Optimizer States

2025/09/27 by Aman Gupta, Gupta, Aman, Rafael Celente +17
Engineering · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Muon and positron interactions and applications #Neutrino Physics Research #Particle Detector Development and Performance

paper · pdf · doi:10.48550/arxiv.2509.23106

openalex publication_date 2025/09/27 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

The Muon optimizer, based on matrix orthogonalization, has recently shown faster convergence and better computational efficiency over AdamW in LLM pre-training. However, the memory overhead of maintaining high-precision optimizer states remains a challenge for large-scale deployment. In this paper, we introduce the 8-bit Muon optimizer using blockwise quantization. In extensive Chinchilla-optimal experiments on pre-training models of up to 2.7B in size and fine-tuning them for instruction following, we demonstrate that 8-bit Muon achieves parity with Muon in terms of validation loss and downstream benchmarks, while achieving up to a 62% reduction in optimizer state footprint. Crucially, we show that Muon's update mechanism is uniquely compatible with a simple linear quantization scheme, bypassing the complex dynamic scaling required for quantized AdamW. We supplement our empirical findings with a theoretical analysis of Muon's robustness to quantization noise.

Citations

• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
• Kimi K2: Open Agentic Intelligence
• MuLoCo: Muon is a practical inner optimizer for DiLoCo
• Practical Efficiency of Muon for Pretraining
• Dion: Distributed Orthonormalized Updates
• Muon is Scalable for LLM Training
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Modular Duality in Deep Learning
• Old Optimizer, New Norm: An Anthology
• SOAP: Improving and Stabilizing Shampoo using Adam
• The Llama 3 Herd of Models
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• LoRA Learns Less and Forgets Less
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• DiLoCo: Distributed Low-Communication Training of Language Models
• QuantEase: Optimization-based Quantization for Language Models
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Training Compute-Optimal Large Language Models
• Training Verifiers to Solve Math Word Problems
• 8-bit Optimizers via Block-wise Quantization
• Evaluating Large Language Models Trained on Code
• LoRA: Low-Rank Adaptation of Large Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Massive Multitask Language Understanding
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• The LAMBADA dataset: Word prediction requiring a broad discourse context
• Deep Residual Learning for Image Recognition
• Adam: A Method for Stochastic Optimization

Related

• MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
• Muon is Scalable for LLM Training
• Beyond the Ideal: Analyzing the Inexact Muon Update
• Practical Efficiency of Muon for Pretraining
• Muon: Training and Trade-offs with Latent Attention and MoE