Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

2026/06/30 by Xin Gao, Xingming Xu
Computer Science · #cs.AI #cs.CL

paper · pdf

13 pages, 4 figures

arxiv created 2026/07/28 · arxiv updated 2026/07/30

Abstract

Transformer architectures form the foundation of modern natural language processing, making it crucial to address the efficiency and scalability limitations of the standard QKV attention mechanism. The Key-Value (KV) cache is a major bottleneck during long-context inference. We propose Keyless Attention, a novel attention mechanism that introduces a dedicated value-space routing projection to replace the conventional key projection, thereby eliminating key representations from the attention computation. This design yields a Value-Only Cache that reduces KV-cache memory and access overhead by 50% compared with standard attention while improving decode throughput. Experiments across five models and four architectures show that Keyless Attention matches or outperforms standard QKV attention in perplexity on four of five models. Furthermore, it achieves competitive performance on downstream evaluation benchmarks while consistently reducing KV-cache memory by 50%. Ablation studies on attention factorization depth further demonstrate that the dedicated value-space routing projection is a key component of the proposed architecture, with Keyless Attention outperforming KV-sharing methods that do not employ this routing mechanism.

Citations

• Gemma 4 Technical Report
• Do Transformers Need Three Projections? Systematic Study of QKV Variants
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
• TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
• ThinK: Thinner Key Cache by Query-Driven Pruning
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
• Qwen2 Technical Report
• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Efficiently Scaling Transformer Inference
• PaLM: Scaling Language Modeling with Pathways
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Rethinking Attention with Performers
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Conv-Linformer: Boosting Linformer's Performance with Convolution in Small-Scale Settings
• Longformer: The Long-Document Transformer
• Fast Transformer Decoding: One Write-Head is All You Need
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Generating Long Sequences with Sparse Transformers
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Crowdsourcing Multiple Choice Science Questions
• Attention Is All You Need
• Pointer Sentinel Mixture Models
• A Corpus and Evaluation Framework for Deeper Understanding of\n Commonsense Stories
• Effective Approaches to Attention-based Neural Machine Translation
• Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA

Related

• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
• MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
• TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
• Lossless KV Cache Compression to 2%
• Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads