Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference

2025/11/19 by Kexin Chu, Chu, Kexin, Dawei Xiang +9 · 1 voice · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Performance (cs.PF) #cs.AI #cs.LG #cs.PF

paper · pdf · doi:10.48550/arxiv.2511.15015

arxiv published 2025/11/19 · arxiv updated 2026/02/06

Abstract

Mixture-of-Experts (MoE) has become a practical architecture for scaling LLM capacity while keeping per-token compute modest, but deploying MoE models on a single, memory-limited GPU remains difficult because expert weights dominate the HBM footprint. Existing expert offloading and prefetching systems reduce the resident set, yet they often pay expert-loading costs on the critical path when activation becomes dense. Post-training quantization (PTQ) lowers the footprint without transfers, but prevailing pipelines fix expert bit-widths offline and assume routing remains stable, even though MoE expert utilization is heavy-tailed and the hot set can shift across workloads. We present DynaExq, a runtime-aware mixed-precision serving system that treats single-GPU MoE inference under a hard HBM envelope as an online, budget-constrained precision allocation problem. The key insight is to keep the experts that dominate runtime traffic resident at higher precision, while maintaining a low-precision fallback for the remaining experts, so the system can reduce transfer volume and avoid the waiting latency that limits offloading and prefetching under dense activation. DynaExq estimates long-horizon expert hotness from router traces, selects a per-layer high-precision resident set via a budget-feasible top-n rule, and applies promotions and demotions asynchronously through stable expert handles so the forward pass always executes on a fully materialized expert version. Across Qwen3-MoE-30B/80B and six benchmarks, DynaExq improves accuracy over static PTQ on Qwen3-80B (73.09% to 77.57%) under comparable device-memory budgets and achieves up to 2.73x higher throughput than offloading/prefetch baselines at batch size 32.

Citations

• ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
• MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
• ProMoE: Fast MoE-based LLM Serving using Proactive Caching
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• Fast Inference of Mixture-of-Experts Language Models with Offloading
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
• Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
• SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget
• Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Mixture-of-Experts with Expert Choice Routing
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
• Training Verifiers to Solve Math Word Problems
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• Pointer Sentinel Mixture Models

Cited by

• HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
• Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning

Discussions

• Kexin Chu, Dawei Xiang, Zixu Shen, Yiwei Yang, Zecheng Liu, Wei Zhang: Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference https://arxiv.org/abs/2511.15015 https://arxiv.org/pdf/2511 [bsky, 0 points, 0 comments]

Related

• Mixture of Experts with Mixture of Precisions for Tuning Quality of Service
• MoPEQ: Mixture of Mixed Precision Quantized Experts
• Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
• EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
• Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems