Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts

2025/08/01 by Yang Jin, Yang, Jin, Qiong Wu +9 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Big Data and Digital Economy #Distributed #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Mobile Crowdsensing and Crowdsourcing #Multiagent Systems (cs.MA) #Networking and Internet Architecture (cs.NI) #Parallel #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2508.00234

openalex publication_date 2025/08/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities, leading to a significant increase in user demand for LLM services. However, cloud-based LLM services often suffer from high latency, unstable responsiveness, and privacy concerns. Therefore, multiple LLMs are usually deployed at the network edge to boost real-time responsiveness and protect data privacy, particularly for many emerging smart mobile and IoT applications. Given the varying response quality and latency of LLM services, a critical issue is how to route user requests from mobile and IoT devices to an appropriate LLM service (i.e., edge LLM expert) to ensure acceptable quality-of-service (QoS). Existing routing algorithms fail to simultaneously address the heterogeneity of LLM services, the interference among requests, and the dynamic workloads necessary for maintaining long-term stable QoS. To meet these challenges, in this paper we propose a novel deep reinforcement learning (DRL)-based QoS-aware LLM routing framework for sustained high-quality LLM services. Due to the dynamic nature of the global state, we propose a dynamic state abstraction technique to compactly represent global state features with a heterogeneous graph attention network (HAN). Additionally, we introduce an action impact estimator and a tailored reward function to guide the DRL agent in maximizing QoS and preventing latency violations. Extensive experiments on both Poisson and real-world workloads demonstrate that our proposed algorithm significantly improves average QoS and computing resource efficiency compared to existing baselines.

Citations

• GraphRouter: A Graph-based Router for LLM Selections
• Eagle: Efficient Training-Free Router for Multi-LLM Inference
• Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models
• RouteLLM: Learning to Route LLMs with Preference Data
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• WDMoE: Wireless Distributed Large Language Models with Mixture of Experts
• Octopus v4: Graph of language models
• Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
• Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
• RouterBench: A Benchmark for Multi-LLM Routing System
• On Protecting the Data Privacy of Large Language Models (LLMs): A Survey
• Toward Scalable Generative AI via Mixture of Experts in Mobile Edge Networks
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Splitwise: Efficient generative LLM inference using phase splitting
• Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models
• EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge
• Large Language Model Routing with Benchmark Datasets
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• S³: Increasing GPU Utilization during Generative Inference for Higher Throughput
• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
• TorchRL: A data-driven decision-making library for PyTorch
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Enabling AI-Generated Content (AIGC) Services in Wireless Edge Networks
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
• BERTScore: Evaluating Text Generation with BERT
• Fast Graph Representation Learning with PyTorch Geometric
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Clipper: A Low-Latency Online Prediction Serving System
• TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
• BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
• Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• The Nonstochastic Multiarmed Bandit Problem

Cited by

• A Synergy of Computing Power Networks and Low-Altitude Economy Intelligent Communications: Challenges, Design Principles, and Research Directions

Related

• VELO: A Vector Database-Assisted Cloud-Edge Collaborative LLM QoS Optimization Framework
• LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters
• Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading
• Learned Best-Effort LLM Serving
• Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRL