Maximum Score Routing For Mixture-of-Experts

2025/08/18 by Bowen Dong, Dong, Bowen, Yaohui Sun +10 · 1 citation
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Computation and Language (cs.CL) #Distributed Sensor Networks and Detection Algorithms #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2508.12801

openalex publication_date 2025/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Routing networks in sparsely activated mixture-of-experts (MoE) dynamically allocate input tokens to top-k experts through differentiable sparse transformations, enabling scalable model capacity while preserving computational efficiency. Traditional MoE networks impose an expert capacity constraint to ensure GPU-friendly computation. However, this leads to token dropping when capacity is saturated and results in low hardware efficiency due to padding in underutilized experts. Removing the capacity constraint, in turn, compromises load balancing and computational efficiency. To address these issues, we propose Maximum Score Routing (MaxScore), a novel MoE routing paradigm that models routing as a minimum-cost maximum-flow problem and integrates a SoftTopk operator. MaxScore resolves the fundamental limitations of iterative rerouting and optimal transport formulations, achieving lower training losses and higher evaluation scores at equivalent FLOPs compared to both constrained and unconstrained baselines. Implementation details and experimental configurations can be obtained from \hrefhttps://github.com/dongbw18/MaxScore.gitMaxScore.

Citations

• ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
• OLMoE: Open Mixture-of-Experts Language Models
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• LLaMA: Open and Efficient Foundation Language Models
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
• Mixture-of-Experts with Expert Choice Routing
• ST-MoE: Designing Stable and Transferable Sparse Expert Models
• Unified Scaling Laws for Routed Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• GLU Variants Improve Transformer
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Root Mean Square Layer Normalization
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Sparse Sequence-to-Sequence Models
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Crowdsourcing Multiple Choice Science Questions
• Attention Is All You Need
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• The LAMBADA dataset: Word prediction requiring a broad discourse context
• Training Deep Nets with Sublinear Memory Cost
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Conditional Computation in Neural Networks for faster models
• Learning Factored Representations in a Deep Mixture of Experts

Cited by

• GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
• Mixture of Experts in Large Language Models

Related

• SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
• Mixture-of-Experts with Expert Choice Routing
• From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing
• Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
• Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts