KVDirect: Distributed Disaggregated LLM Inference

2024/12/13 by Shiyang Chen, Chen, Shiyang, Jiang, Rain +13 · 4 citations
Computer Science · #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Parallel #Performance (cs.PF) #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2501.14743

openalex publication_date 2024/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to improving hardware utilization and service quality. However, due to inefficient inter-node communication, existing systems restrict disaggregated inference to a single node, limiting resource allocation flexibility and reducing service capacity. This paper introduces KVDirect, which optimizes KV cache transfer to enable a distributed disaggregated LLM inference. KVDirect achieves this through the following contributions. First, we propose a novel tensor-centric communication mechanism that reduces the synchronization overhead in traditional distributed GPU systems. Second, we design a custom communication library to support dynamic GPU resource scheduling and efficient KV cache transfer. Third, we introduce a pull-based KV cache transfer strategy that reduces GPU resource idling and improves latency. Finally, we implement KVDirect as an open-source LLM inference framework. Our evaluation demonstrates that KVDirect reduces per-request latency by 55% compared to the baseline across diverse workloads under the same resource constraints.

Cited by

• Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
• BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
• Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

Related

• KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
• FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling
• From Attention to Disaggregation: Tracing the Evolution of LLM Inference
• LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference