SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference

2025/07/09 by Qian Chen, Xianhao Chen, Chen, Qian +3 · 3 citations
Computer Science · #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Networking and Internet Architecture (cs.NI) #Parallel #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2507.06567

openalex publication_date 2025/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Mixture-of-Experts (MoE) models improve the scalability of large language models (LLMs) by activating only a small subset of relevant experts per input. However, the sheer number of expert networks in an MoE model introduces a significant storage/memory burden for an edge device. To address this challenge, we consider a scenario where experts are dispersed across an edge network for distributed inference. Based on the popular Top-K expert selection strategy, we formulate a latency minimization problem by optimizing expert caching on edge servers under storage constraints. When K=1, the problem reduces to a monotone submodular maximization problem with knapsack constraints, for which we design a greedy-based algorithm with a (1 - 1/e)-approximation guarantee. For the general case where K≥1, expert co-activation within the same MoE layer introduces non-submodularity, which renders greedy methods ineffective. To tackle this issue, we propose a successive greedy decomposition method to decompose the original problem into a series of subproblems, with each being solved by a dynamic programming approach. Furthermore, we design an accelerated algorithm based on the max-convolution technique to obtain the approximate solution with a provable guarantee in polynomial time. Simulation results on various MoE models demonstrate that our method significantly reduces inference latency compared to existing baselines.

Citations

• HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
• Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
• Semantic Sleuth: Identifying Ponzi Contracts via Large Language Models
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
• A Review on Edge Large Language Models: Design, Execution, and Applications
• On-Device Language Models: A Comprehensive Review
• AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Merging
• Mobile Edge Intelligence for Large Language Models: A Contemporary Survey
• SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• TrimCaching: Parameter-sharing AI Model Caching in Wireless Edge Networks
• TrimCaching: Parameter-sharing Edge Caching for AI Model Downloading
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
• Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
• Mixtral of Experts
• Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
• EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
• Wireless Edge Content Broadcast via Integrated Terrestrial and Non-terrestrial Networks
• LLaMA: Open and Efficient Foundation Language Models
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Hash Layers For Large Sparse Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Simple Deterministic Approximation for Submodular Multiple Knapsack Problem
• Scaling Laws for Neural Language Models
• A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
• Two-level Transmission Scheme for Cache-enabled Fog Radio Access Networks
• Capacitated Dynamic Programming: Faster Knapsack and Graph Algorithms
• Greed is Still Good: Maximizing Monotone Submodular+Supermodular Functions
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling

Cited by

• SLIDE: Simultaneous Model Downloading and Inference at the Wireless Network Edge
• OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
• TrimMoE A communication aware and adaptive depth framework for distributed edge inference

Related

• Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
• CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edge
• MoE²: Optimizing Collaborative Inference for Edge Large Language Models
• OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
• Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism