PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services

2024/05/23 by Zheming Yang, Yuanhao Yang, Yang, Zheming +9 · 5 citations
Computer Science · Decision Sciences · #Distributed #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Networking and Internet Architecture (cs.NI) #Parallel #Privacy-Preserving Technologies in Data #Scientific Computing and Data Management #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2405.14636

openalex publication_date 2024/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the rapid growth in the number of large language model (LLM) users, it is difficult for bandwidth-constrained cloud servers to simultaneously process massive LLM services in real-time. Recently, edge-cloud infrastructures have been used to improve the processing efficiency of large-scale LLM services. However, the diversity of task requirements and the dynamics of resources pose great challenges to inference scheduling, leading to the wastage of many resources. In this paper, we present PerLLM, a personalized inference scheduling framework with edge-cloud collaboration designed for diverse LLM services. For the complexity of multiple constraints and the decision-making process of edge-cloud collaboration, we integrate the upper confidence bound algorithm based on the constraint satisfaction mechanism in PerLLM. For diverse LLM services, PerLLM can optimize service scheduling and resource allocation solutions within the edge-cloud infrastructure to meet processing time requirements while minimizing energy costs. Experimental results from different model deployments show that PerLLM can effectively meet the processing time requirements of personalized services. Compared to other methods, PerLLM achieves 2.2x, 2.1x, and 1.6x throughput and reduces the energy cost by more than 50%.

Cited by

• Toward Sustainability-Aware LLM Inference on Edge Clusters
• Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization
• Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detection
• SAEC: Scene-Aware Enhanced Edge-Cloud Collaborative Industrial Vision Inspection with Multimodal LLM
• MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inference

Related

• Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
• HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference
• EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
• Cloud Native System for LLM Inference Serving
• CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration