ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs

2025/03/13 by Xin Liu, Xudong Wang, Liu, Xin +5 · 7 voices · 2 citations
Computer Science · #Big Data and Digital Economy #Natural Language Processing Techniques #Parallel Computing and Optimization Techniques #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2503.10714

openalex publication_date 2025/03/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The linear growth of key-value (KV) cache memory and quadratic computational in attention mechanisms complexity pose significant bottlenecks for large language models (LLMs) in long-context processing. While existing KV cache optimization methods address these challenges through token pruning or feature merging, they often incur irreversible information loss or require costly parameter retraining. To this end, we propose ZSMerge, a dynamic KV cache compression framework designed for efficient cache management, featuring three key operations: (1) fine-grained memory allocation guided by multi-dimensional token importance metrics at head-level granularity, (2) a residual merging mechanism that preserves critical context through compensated attention scoring, and (3) a zero-shot adaptation mechanism compatible with diverse LLM architectures without requiring retraining. ZSMerge significantly enhances memory efficiency and inference speed with negligible performance degradation across LLMs. When applied to LLaMA2-7B, it demonstrates a 20:1 compression ratio for key-value cache retention (reducing memory footprint to 5% of baseline) while sustaining comparable generation quality, coupled with triple throughput gains at extreme 54k-token contexts that eliminate out-of-memory failures. The code is available at https://github.com/SusCom-Lab/ZSMerge.

Citations

• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• Yi: Open Foundation Models by 01.AI
• Efficient Streaming Language Models with Attention Sinks
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Lost in the Middle: How Language Models Use Long Contexts
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Attention Is All You Need

Cited by

• Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

Discussions

• Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs #HackerNews https://arxiv.org/abs/2503.10714 [bsky, 1 points, 0 comments]
• Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs https://arxiv.org/abs/2503.10714 (https://news.ycombinator.com/item?id=43496244) [bsky, 0 points, 0 comments]
• Parameter-free KV cache compression for memory-efficient long-context LLMs [bsky, 0 points, 0 comments]
• Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs https://arxiv.org/abs/2503.10714 [bsky, 0 points, 0 comments]
• Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• Parameter-Free KV Cache Compression for Memory-Efficient Long-Context LLMs https://arxiv.org/abs/2503.10714 (https://news.ycombinator.com/item?id=43496244) [bsky, 0 points, 0 comments]
• Parameter-free KV cache compression for memory-efficient long-context LLMs https:// arxiv.org/abs/2503.10714 # arxiv # llm # llms [mastodon, 0 points, 0 comments]

Related

• KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
• KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
• KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
• ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
• KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference