A Method for Building Large Language Models with Predefined KV Cache Capacity

2024/11/24 by Zhonghua Yi, Yi, Zhonghua, Guangda Niu +6
Computer Science · #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2411.15785

openalex publication_date 2024/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces a novel approach, the Bounded-Cache Transformer (BCT), for building large language models with a predefined Key-Value (KV) cache capacity. The BCT addresses the excessive memory consumption issue in traditional KV caches by implementing a bounded-length KV cache, which is particularly suitable for the attention layers in Transformer decode-only architectures. By dynamically updating the key-value vector sequences, the BCT achieves efficient inference within limited cache capacity, significantly reducing memory usage while maintaining model performance and system throughput. Experimental results demonstrate that the BCT significantly reduces memory usage while maintaining the model's inference quality, offering a new solution for efficient inference in large language models.

Related

• Layer-Condensed KV Cache for Efficient Inference of Large Language Models
• CORM: Cache Optimization with Recent Message for Large Language Model Inference
• TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance