KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider

2025/06/03 by Jiahao Wang, Wang, Jiahao, Xingda Wei +13 · 12 citations
Computer Science · #Advanced Data Storage Technologies #Artificial Intelligence (cs.AI) #Caching and Content Delivery #Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2506.02634

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV$) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV$ caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV$ workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV$ reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity.

Cited by

• Role-Based Fault Tolerance System for LLM RL Post-Training
• MEPIC: Memory Efficient Position Independent Caching for LLM Serving
• ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
• WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
• A Systematic Characterization of LLM Inference on GPUs
• On 10x Better Scalability: KV Stores Scale Up KV Cache
• FairBatching: Fairness-Aware Batch Formation for LLM Inference
• Toward Robust and Efficient ML-Based GPU Caching for Modern Inference
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
• TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
• Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

Related

• Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
• KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
• Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
• Compute Or Load KV Cache? Why Not Both?
• KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse