Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Decoder-Only Transformers

2025/10/27 by Karbevski, Marko, Mijoski, Antonij
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.23912

Abstract

The Query, Key, Value weight triplet is a building block of current attention mechanisms in state-of-the-art LLMs. We theoretically investigate whether this triplet can be reduced, proving under simplifying assumptions that the Query weights are redundant, thereby reducing the number of non-embedding/lm-head parameters by over 8%. We validate the theory on full-complexity GPT-3 small architectures (with layer normalization, skip connections, and weight decay) trained from scratch, demonstrating that the reduced model achieves comparable validation loss to standard baselines. These findings motivate the investigation of the Query weight redundancy at scale.

Citations

• mHC: Manifold-Constrained Hyper-Connections
• Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability
• Qwen3 Technical Report
• Always Skip Attention
• Hyper-Connections
• You can remove GPT2's LayerNorm by fine-tuning
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment
• A mathematical perspective on Transformers
• Simplifying Transformer Blocks
• Why Do We Need Weight Decay in Modern Deep Learning?
• From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference
• QLoRA: Efficient Finetuning of Quantized LLMs
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• LLM-Pruner: On the Structural Pruning of Large Language Models
• The emergence of clusters in self-attention dynamics
• Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Training Compute-Optimal Large Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Query-Key Normalization for Transformers
• Rethinking Attention with Performers
• Efficient Transformers: A Survey
• Conv-Linformer: Boosting Linformer's Performance with Convolution in Small-Scale Settings
• Language Models are Few-Shot Learners
• Are Transformers universal approximators of sequence-to-sequence functions?
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fast Transformer Decoding: One Write-Head is All You Need
• Root Mean Square Layer Normalization
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Deep Equilibrium Models
• Parameter-Efficient Transfer Learning for NLP
• Visualizing the Loss Landscape of Neural Nets
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Using the Output Embedding to Improve Language Models
• Layer Normalization
• Gaussian Error Linear Units (GELUs)
• Identity Mappings in Deep Residual Networks
• Deep Residual Learning for Image Recognition
• On the Number of Linear Regions of Deep Neural Networks
• Adaptive Mixtures of Local Experts

Related

• MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
• Weighted Grouped Query Attention in Transformers
• Quantize What Counts: More for Keys, Less for Values
• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
• Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers