Interleaved Head Attention

2026/02/24 by Sai Surya Duvvuri, Chanakya Ekbote, Rachit Bansal +6 · 1 voice
Computer Science · #Computational complexity theory #Context (archaeology) #Head (geology) #Mixing (physics) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Overhead (engineering) #Polynomial #Scaling #Task (project management) #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.2602.21371

openalex publication_date 2026/02/24 · arxiv published 2026/02/24 · arxiv updated 2026/02/24 · openalex created_date 2026/02/27 · openalex updated_date 2026/07/28

Abstract

Multi-Head Attention (MHA) is the core computational primitive underlying modern Large Language Models (LLMs). However, MHA suffers from a fundamental linear scaling limitation: H attention heads produce exactly H independent attention matrices, with no communication between heads during attention computation. This becomes problematic for multi-step reasoning, where correct answers depend on aggregating evidence from multiple parts of the context and composing latent token-to-token relations over a chain of intermediate inferences. To address this, we propose Interleaved Head Attention (IHA), which enables cross-head mixing by constructing P pseudo-heads per head (typically P=H), where each pseudo query/key/value is a learned linear combination of all H original queries, keys and values respectively. Interactions between pseudo-query and pseudo-key heads induce up to P² attention patterns per head with modest parameter overhead O(H²P). We provide theory showing improved efficiency in terms of number of parameters on the synthetic Polynomial task (IHA uses Θ(√(k)n²) parameters vs. Θ(kn²) for MHA) and on the synthetic order-sensitive CPM-3 task (IHA uses \lceil√N_max\rceil heads vs. N_max for MHA). On real-world benchmarks, IHA improves Multi-Key retrieval on RULER by 10-20% (4k-16k) and, after fine-tuning for reasoning on OpenThoughts, improves GSM8K by 5.8% and MATH-500 by 2.8% (Majority Vote) over full attention.

Citations

• Knocking-Heads Attention
• Fast and Simplex: 2-Simplicial Attention in Triton
• Multi-Token Attention
• Reasoning with Latent Thoughts: On the Power of Looped Transformers
• Strassen Attention, Split VC Dimension and Compositionality in Transformers
• The Llama 3 Herd of Models
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations
• Representational Strengths and Limitations of Transformers
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• A Piece-wise Polynomial Filtering Approach for Graph Neural Networks
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Talking-Heads Attention
• Fast Transformer Decoding: One Write-Head is All You Need
• Logic and the 2-Simplicial Transformer
• Mixed Precision Training
• Attention Is All You Need
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

Discussions

• 15/ Interleaved Head Attention New Transformer architecture where attention heads share information instead of staying isolated. Claims improved reasoning and compositionality. arXiv: https://arxiv.or [bsky, 1 points, 1 comments]

Related

• Knocking-Heads Attention
• DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
• MoH: Multi-Head Attention as Mixture-of-Head Attention
• Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention
• Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA