MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

2024/11/18 by Shiyi Cao, Shu Liu, Cao, Shiyi +15 · 21 citations
Computer Science · #Artificial Intelligence (cs.AI) #Cloud Computing and Resource Management #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2411.11217

openalex publication_date 2024/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Efficient deployment of large language models, particularly Mixture of Experts (MoE), on resource-constrained platforms presents significant challenges, especially in terms of computational efficiency and memory utilization. The MoE architecture, renowned for its ability to increase model capacity without a proportional increase in inference cost, greatly reduces the token generation latency compared with dense models. However, the large model size makes MoE models inaccessible to individuals without high-end GPUs. In this paper, we propose a high-throughput MoE batch inference system, that significantly outperforms past work. MoE-Lightning introduces a novel CPU-GPU-I/O pipelining schedule, CGOPipe, with paged weights to achieve high resource utilization, and a performance model, HRM, based on a Hierarchical Roofline Model we introduce to help find policies with higher throughput than existing systems. MoE-Lightning can achieve up to 10.3x higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB). When the theoretical system throughput is bounded by the GPU memory, MoE-Lightning can reach the throughput upper bound with 2-3x less CPU memory, significantly increasing resource utilization. MoE-Lightning also supports efficient batch inference for much larger MoEs (e.g., Mixtral 8x22B and DBRX) on multiple low-cost GPUs (e.g., 2-4 T4).

Cited by

• Decoding the Skew: Distribution-Aware MoE Inference with Adaptive Kernel Dispatch
• Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
• Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation
• Janus: Disaggregating Attention and Experts for Scalable MoE Inference
• Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems
• OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
• Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles
• FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
• ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
• HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission
• Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference
• PreScope: Unleashing the Power of Prefetching for Resource-Constrained MoE Inference
• ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale
• ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
• Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
• Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
• Enabling MoE on the Edge via Importance-Driven Expert Scheduling
• BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs
• Utility-Driven Speculative Decoding for Mixture-of-Experts
• Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs
• L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models

Related

• MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
• MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching
• Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
• Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
• HarMoEny: Efficient Multi-GPU Inference of MoE Models