ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

2025/01/25 by Shangqian Gao, Gao, Shangqian, Ting Hua +20 · 5 citations
Computer Science · Engineering · #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Speech and dialogue systems #Traffic Prediction and Management Techniques

paper · pdf · doi:10.48550/arxiv.2501.15316

openalex publication_date 2025/01/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

Cited by

• HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
• StructPrune: Structured Global Pruning asymptotics with O(√(N)) GPU Memory
• ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
• Visualising Information Flow in Word Embeddings with Diffusion Tensor Imaging

Related

• Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
• Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
• DLP: Dynamic Layerwise Pruning in Large Language Models
• GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching
• MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models