Exploiting the Experts: Unauthorized Compression in MoE-LLMs

2025/11/22 by Neogi, Pinaki Prasad Guha, Mohammadshirazi, Ahmad, Kulshrestha, Dheeraj +1 · 1 citation
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data

paper · doi:10.48550/arxiv.2511.19480

openalex publication_date 2025/11/22 · openalex created_date 2025/11/28 · openalex updated_date 2026/07/28

Abstract

Mixture-of-Experts (MoE) architectures are increasingly adopted in large language models (LLMs) for their scalability and efficiency. However, their modular structure introduces a unique vulnerability: adversaries can attempt to compress or repurpose models by pruning experts and cheaply fine-tuning the remainder, effectively bypassing licensing and security constraints. In this paper, we systematically study the prunability of MoE-LLMs under task-specific usage. We first develop an expert attribution framework that identifies the subset of experts most responsible for a given task, then evaluate the performance trade-offs of pruning and re-aligning these experts using active learning-driven fine-tuning. Our findings reveal a critical knowledge loss--recovery trade-off: while certain experts can be isolated to retain task accuracy, significant degradation occurs without targeted re-alignment. Based on this analysis, we propose defense strategies that aim to make MoE models harder to compress and fine-tune without authorization, including entangled expert training and selective fine-tuning protocols that resist unauthorized adaptation. By positioning expert pruning as both a threat vector and a defense target, this work highlights the dual-use nature of MoE modularity and provides the first systematic evaluation framework for secure specialization of MoE-LLMs.

Citations

• MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
• Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
• Mixtral of Experts
• A Simple and Effective Pruning Approach for Large Language Models
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• Task-Specific Expert Pruning for Sparse Mixture-of-Experts
• Mixture-of-Experts with Expert Choice Routing
• ST-MoE: Designing Stable and Transferable Sparse Expert Models
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• Hash Layers For Large Sparse Models
• BASE Layers: Simplifying Training of Large, Sparse Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Transformer Feed-Forward Layers Are Key-Value Memories
• Extracting Training Data from Large Language Models
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• High Accuracy and High Fidelity Extraction of Neural Networks
• The State of Sparsity in Deep Neural Networks
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Pointer Sentinel Mixture Models
• Pruning Filters for Efficient ConvNets
• Stealing Machine Learning Models via Prediction APIs
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning both Weights and Connections for Efficient Neural Networks

Cited by

• DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA

Related

• Mixture Compressor for Mixture-of-Experts LLMs Gains More
• Efficiently Editing Mixture-of-Experts Models with Compressed Experts
• BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Experts
• Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
• Unveiling Super Experts in Mixture-of-Experts Large Language Models