EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization

2025/06/16 by Zhongqian Lv, Tianyi Zhao, Fu, Zhongqian +12 · 1 citation
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #Context-Aware Activity Recognition Systems #FOS: Computer and information sciences #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2506.13329

openalex publication_date 2025/06/16 · openalex created_date 2025/10/20 · openalex updated_date 2026/07/28

Abstract

Mixture-of-Experts (MoE) models enable scalable computation and performance in large-scale deep learning but face quantization challenges due to sparse expert activation and dynamic routing. Existing post-training quantization (PTQ) methods fail to address activation outliers, routing instability, and sparse expert calibration, leading to significant performance degradation. To address this, we propose EAQuant, a PTQ framework tailored for MoE architectures. Our method introduces three expert-aware innovations: (1) smoothing aggregation to suppress activation outliers, (2) routing consistency alignment to preserve expert selection post-quantization, and (3) calibration data balance to optimize sparsely activated experts. These strategies collectively enable robust, high-precision quantization of MoE models under ultra-low-bit constraints.Extensive experiments across several extreme quantization settings (e.g., W4A4/W3A4/W3A3/W2A4) demonstrate that EAQuant significantly outperforms existing methods, achieving average accuracy improvements of 1.15 - 13.81% across three diverse MoE architectures, with particularly pronounced gains in reasoning tasks and robust performance retention under aggressive quantization. By integrating these innovations, EAQuant establishes a new state-of-the-art for high-precision, efficient MoE model compression.Our code is available at https://github.com/darren-fzq1/EAQuant.

Citations

• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
• OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
• A Survey on Inference Optimization Techniques for Mixture of Experts Models
• Mixture of Experts with Mixture of Precisions for Tuning Quality of Service
• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
• SpinQuant: LLM quantization with learned rotations
• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
• Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Pointer Sentinel Mixture Models

Cited by

• VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse

Related

• QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
• MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
• Ban&Pick: Ehancing Performance and Efficiency of MoE-LLMs via Smarter Routing
• Dense Backpropagation Improves Training for Sparse Mixture-of-Experts