Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

2026/05/12 by Kexuan Shi, Hanxuan Li, Zeju Qiu +3 · 1 voice
Computer Science · Mathematics · #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2605.12492

Abstract

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Citations

• The Path Not Taken: RLVR Provably Learns Off the Principals
• NorMuon: Making Muon more efficient and scalable
• gpt-oss-120b & gpt-oss-20b Model Card
• Kimi K2: Open Agentic Intelligence
• Muon Optimizes Under Spectral Norm Constraints
• Reparameterized LLM Training via Orthogonal Equivalence Transformation
• The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm
• DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
• Muon is Scalable for LLM Training
• Training Deep Learning Models with Norm-Constrained LMOs
• The Curse of Depth in Large Language Models
• Modular Duality in Deep Learning
• Methods of improving LLM training stability
• Old Optimizer, New Norm: An Anthology
• SOAP: Improving and Stabilizing Shampoo using Adam
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• LoRA Learns Less and Forgets Less
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Magicoder: Empowering Code Generation with OSS-Instruct
• A Spectral Condition for Feature Learning
• Small-scale proxies for large-scale Transformer training instabilities
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Controlling Text-to-Image Diffusion by Orthogonal Finetuning
• Scaling Vision Transformers to 22 Billion Parameters
• Solving Quantitative Reasoning Problems with Language Models
• Training Compute-Optimal Large Language Models
• Orthogonalising gradients to speed up neural network optimisation
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• CogView: Mastering Text-to-Image Generation via Transformers
• Learning with Hyperspherical Uniformity
• Feature Learning in Infinite-Width Neural Networks
• Query-Key Normalization for Transformers
• Orthogonal Over-Parameterized Training
• GLU Variants Improve Transformer
• On Layer Normalization in the Transformer Architecture
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Root Mean Square Layer Normalization
• Regularizing Neural Networks via Minimizing Hyperspherical Energy
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Riemannian Adaptive Optimization Methods
• Learning towards Minimum Hyperspherical Energy
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Spectral Normalization for Generative Adversarial Networks
• On the Convergence of Adam and Beyond
• Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
• Spectrally-normalized margin bounds for neural networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• Layer Normalization
• Optimization Methods for Large-Scale Machine Learning
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ADADELTA: An Adaptive Learning Rate Method
• A Stochastic Approximation Method
• Some methods of speeding up the convergence of iteration methods

Discussions

• Pion proposes a spectrum-preserving optimizer for LLMs: instead of additive updates, it applies orthogonal transformations that keep singular values fixed. Competitive performance, stable training, an [bsky, 0 points, 0 comments]

Related

• Reparameterized LLM Training via Orthogonal Equivalence Transformation
• ROOT: Robust Orthogonalized Optimizer for Neural Network Training
• REG: A Regularization Optimizer for Robust Training Dynamics
• Muon is Scalable for LLM Training
• PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer