PQCache: Product Quantization-based KVCache for Long Context LLM Inference

2024/07/01 by Hailin Zhang, Xiaodong Ji, Zhang, Hailin +13 · 19 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2407.12820

openalex publication_date 2024/07/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques prevalent in the data management community, we consider the storage and retrieval of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, we use PQ codes and centroids to approximately identify important preceding tokens, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments demonstrate that PQCache achieves both effectiveness and efficiency, with 4.60% score improvement over existing methods on InfiniteBench and low system latency in both prefilling and decoding.

Cited by

• A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference
• OrchANN: Hierarchical Orchestration for Skewed Out-of-Core Vector Search
• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention
• LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp
• Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
• LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
• ContextPilot: Fast Long-Context Inference via Context Reuse
• Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
• LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• vAttention: Verified Sparse Attention
• Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
• SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
• KV Cache Compression for Inference Efficiency in LLMs: A Review
• SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference

Related

• C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
• Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
• QAQ: Quality Adaptive Quantization for LLM KV Cache
• BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
• KVCrush: Key value cache size-reduction using similarity in head-behaviour