S-LoRA: Serving Thousands of Concurrent LoRA Adapters

2023/11/06 by Ying Sheng, Shiyi Cao, Sheng, Ying +21 · 2 voices · 45 citations
Computer Science · #Adapter (computing) #Advanced Neural Network Applications #CUDA #Cache #Computer network #Computer science #Operating system #Overhead (engineering) #Paging #Parallel Computing and Optimization Techniques #Parallel computing #Scalability #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2311.03285

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services. The code is available at https://github.com/S-LoRA/S-LoRA

Cited by

• InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata
• Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
• JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models
• A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services
• Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs
• Per-Axis Weight Deltas for Frequent Model Updates
• Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
• FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
• CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation
• Adaptive Minds: Empowering Agents with LoRA-as-Tools
• MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint
• VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
• DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
• Decoupling Continual Semantic Segmentation
• Tensorized Clustered LoRA Merging for Multi-Task Interference
• PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models
• LOST: Low-rank and Sparse Pre-training for Large Language Models
• Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices
• On Evaluating Performance of LLM Inference Serving Systems
• LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
• Symbiosis: Multi-Adapter Inference and Fine-Tuning
• MoIRA: Modular Instruction Routing Architecture for Multi-Task Robotics
• EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
• A Survey of LLM Inference Systems
• LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
• Efficient Serving of LLM Applications with Probabilistic Demand Modeling
• PoLAR: Polar-Decomposed Low-Rank Adapter Representation
• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
• InstGenIE: Generative Image Editing Made Efficient with Mask-aware Caching and Scheduling
• HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
• Get Experience from Practice: LLM Agents with Record & Replay
• Enhancing Learned Knowledge in LoRA Adapters Through Efficient Contrastive Decoding on Ascend NPUs
• ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs
• Shared LoRA Subspaces for almost Strict Continual Learning
• MinT: Managed Infrastructure for Training and Serving Millions of LLMs
• Low-Rank Adaptation Redux for Large Models
• Prometheus Mind: Retrofitting Memory to Frozen Language Models
• HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models
• L3: DIMM-PIM Integrated Architecture and Coordination for Scalable Long-Context LLM Inference
• Efficient Split Federated Learning for Large Language Models over Communication Networks
• Activated LoRA: Fine-tuned LLMs for Intrinsics
• PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving

Discussions

• S-LoRA: Serving Concurrent LoRA Adapters [hn, 41 points, 8 comments]
• I would think LoRA's would be more compressed than text so should be faster to load from memory (relative to a prompt) and you get to avoid the FLOP required for doing attention on the additional cont [bsky, 2 points, 1 comments]

Related

• Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
• LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs
• mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs