DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving

2024/03/04 by Strati, Foteini, Mcallister, Sara, Phanishayee, Amar +2 · 13 citations
#Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2403.01876

Abstract

Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. In this paper, we propose DéjàVu, a system to address all these challenges using a versatile and efficient KV cache streaming library (DéjàVuLib). Using DéjàVuLib, we propose and implement efficient prompt-token disaggregation to reduce pipeline bubbles, microbatch swapping for efficient GPU memory management, and state replication for fault-tolerance. We highlight the efficacy of these solutions on a range of large models across cloud deployments.

Cited by

• RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
• MatKV: Trading Compute for Flash Storage in LLM Inference
• Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
• OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
• AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism
• CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation
• Towards Sampling Data Structures for Tensor Products in Turnstile Streams
• SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
• MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems

Related

• Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty
• ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
• KVDirect: Distributed Disaggregated LLM Inference
• FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
• Oneiros: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving