StableMoE: Stable Routing Strategy for Mixture of Experts

2022/04/18 by Damai Dai, Dai, Damai, Dong Li +11 · 15 citations
Computer Science · Physics and Astronomy · #Advanced Graph Neural Networks #Complex Network Analysis Techniques #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2204.08396

openalex publication_date 2022/04/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i.e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference. The routing fluctuation tends to harm sample efficiency because the same input updates different experts but only one is finally used. In this paper, we propose StableMoE with two training stages to address the routing fluctuation problem. In the first training stage, we learn a balanced and cohesive routing strategy and distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the token-to-expert assignment and freeze it for a stable routing strategy. We validate our method on language modeling and multilingual machine translation. The results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.

Cited by

• ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization
• Route Experts by Sequence, not by Token
• Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
• Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
• Expert Merging in Sparse Mixture of Experts with Nash Bargaining
• Toward Efficient Inference Attacks: Shadow Model Sharing via Mixture-of-Experts
• Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
• VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routing
• DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
• Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
• Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
• Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
• Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
• Hierarchical MoE: Continuous Multimodal Emotion Recognition with Incomplete and Asynchronous Inputs
• RelMap: Enhancing Online Map Construction with Class-Aware Spatial Relation and Semantic Priors

Related

• Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
• Mixture-of-Experts with Expert Choice Routing
• S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
• Continual Pre-training of MoEs: How robust is your router?
• Harder Tasks Need More Experts: Dynamic Routing in MoE Models