Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

2025/04/19 by Hang Zhang, Zhang, Hang, Shi, Jiuchen +8 · 6 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Big Data and Digital Economy #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques #Performance (cs.PF)

paper · pdf · doi:10.48550/arxiv.2505.03756

openalex publication_date 2025/04/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.

Citations

• Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
• The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
• Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena
• LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• DoRA: Weight-Decomposed Low-Rank Adaptation
• CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
• LLM in a flash: Efficient Large Language Model Inference with Limited Memory
• Prompt Cache: Modular Attention Reuse for Low-Latency Inference
• Punica: Multi-Tenant LoRA Serving
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Models
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• QLoRA: Efficient Finetuning of Quantized LLMs
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• PaLM: Scaling Language Modeling with Pathways
• LoRA: Low-Rank Adaptation of Large Language Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• GPT-3: Its Nature, Scope, Limits, and Consequences
• Language Models are Few-Shot Learners
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
• Fast Transformer Decoding: One Write-Head is All You Need
• Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
• Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale
• MinT: Managed Infrastructure for Training and Serving Millions of LLMs

Related

• Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters
• Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
• PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models
• A Survey on Large Language Model Acceleration based on KV Cache Management