MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper

2025/08/31 by Zeng, Runjia, Guanqing Sun, Qifan Wang +18 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Topological and Geometric Data Analysis

paper · pdf · doi:10.48550/arxiv.2509.00996

openalex publication_date 2025/08/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Considering deep neural networks as manifold mappers, the pretrain-then-fine-tune paradigm can be interpreted as a two-stage process: pretrain establishes a broad knowledge base, and fine-tune adjusts the model parameters to activate specific neural pathways to align with the target manifold. Although prior fine-tuning approaches demonstrate success, their rigid parameter space limits their ability to dynamically activate appropriate neural pathways, rendering them ill-equipped to adapt flexibly to the diverse and evolving data distributions. In light of this view, we propose a novel approach, Mixture of Expert Prompt Tuning (MEPT), as an effective and efficient manifold-mapping framework. MEPT leverages the Mixture of Experts architecture by integrating multiple prompt experts to adaptively learn diverse and non-stationary data distributions. Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE, achieving notable improvements in mean accuracy (e.g., 1.94%) while significantly reducing activated prompts by 79.25%. The effectiveness of MEPT is further supported by theoretical insights from manifold learning and validated through neural activation pathway visualization results. Our code is avaliable at https://runjia.tech/emnlp_mept/.

Citations

• PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning
• The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models
• The Llama 3 Herd of Models
• SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings
• Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion
• LoRA Dropout as a Sparsity Regularizer for Overfitting Control
• LoRA+: Efficient Low Rank Adaptation of Large Models
• Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy
• E²VPT: An Effective and Efficient Approach for Visual Prompt Tuning
• Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization
• DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
• Unified Vision and Language Prompt Learning
• XPrompt: Exploring the Extreme of Prompt Tuning
• Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Reducing Activation Recomputation in Large Transformer Models
• Prompting Visual-Language Models for Efficient Video Understanding
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• Towards a Unified View of Parameter-Efficient Transfer Learning
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Exploring Visual Engagement Signals for Representation Learning
• An Empirical Study of Training Self-Supervised Vision Transformers
• GPT Understands, Too
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Parameter-Efficient Transfer Learning with Diff Pruning
• AdapterHub: A Framework for Adapting Transformers
• Language Models are Few-Shot Learners
• MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Scaling Laws for Neural Language Models
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Parameter-Efficient Transfer Learning for NLP
• WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
• Dissecting Contextual Word Embeddings: Architecture and Representation
• Exploring the Limits of Weakly Supervised Pretraining
• LDMNet: Low Dimensional Manifold Regularized Neural Networks
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Learning Factored Representations in a Deep Mixture of Experts
• A Fast Learning Algorithm for Deep Belief Nets
• Adaptive Mixtures of Local Experts
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• All You Need is One: Capsule Prompt Tuning with a Single Vector
• Prompt-based Adaptation in Large-scale Vision Models: A Survey
• Probabilistic Token Alignment for Large Language Model Fusion

Related

• Mixture of Neuron Experts
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
• Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models