WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving

2025/12/10 by Chiheng Lou, Shouliang Qi, Lou, Chiheng +14
Computer Science · #Big Data and Digital Economy #Distributed #FOS: Computer and information sciences #GPU cluster #General-purpose computing on graphics processing units #Inference #Key (lock) #Machine Learning (cs.LG) #Natural Language Processing Techniques #Parallel #Parallel Computing and Optimization Techniques #Predictability #Resource (disambiguation) #Supercomputer #Workload #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2512.09472

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/12/10 · openalex created_date 2025/12/12 · openalex updated_date 2026/07/28

Abstract

Deploying multiple models within shared GPU clusters is a key strategy to improve resource efficiency in large language model (LLM) serving. Existing multi-LLM serving systems improve GPU utilization at the cost of degraded inference performance, particularly time-to-first-token (TTFT). We attribute this degradation to the lack of awareness regarding future workload characteristics. In contrast, recent analyses have shown the strong periodicity and long-term predictability of real-world LLM serving workloads. In this paper, we propose one-for-many GPU prewarming, which proactively loads parameters from multiple models onto GPUs based on workload forecasts. These prewarmed weights enable the system to promptly instantiate serving instances upon encountering request bursts. We design and implement WarmServe, a multi-LLM serving system incorporating three key techniques: (1) a model placement algorithm that optimizes prewarming decisions to minimize cross-model prewarming interference, (2) a KV cache reservation strategy that repurposes idle KV cache space on running GPUs for prewarming new models, and (3) an efficient GPU memory switching mechanism for tensor management. Evaluation on real-world datasets shows that WarmServe reduces tail TTFT by up to 50.8× compared to the state-of-the-art autoscaling-based system, while supporting up to 2.5× higher request throughput than the GPU-sharing system.

Citations

• Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs
• Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity
• TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
• KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
• ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production
• Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning
• PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
• HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
• DeepServe: Serverless Large Language Model Serving at Scale
• KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
• GPT-4o System Card
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• The Llama 3 Herd of Models
• Enabling Elastic Model Serving with MultiWorld
• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
• LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
• MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
• BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
• ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
• DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
• Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
• SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inference
• Fast Distributed Inference Serving for Large Language Models
• GPT-4 Technical Report
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• No Provisioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computing
• Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
• Ray: A Distributed Framework for Emerging AI Applications
• Forecasting Sales by Exponentially Weighted Moving Averages

Related

• ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
• DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
• LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
• AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
• PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling