Developmental Atlas of Attention Head Specialization: Spacing, Stranding, and the Capacity Tax of BPE Tokenization

2025/11/21 by Tatsuya Aoyama, Ethan Gotlieb Wilcox, Aoyama, Tatsuya +3 · 1 citation
Social Sciences · Computer Science · #Language and cultural evolution #Topic Modeling #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2511.16893

Abstract

Approximately half of all attention heads in standard BPE models are consumed by mandatory whitespace boundary recovery. Removing these heads degrades performance by +64%, proving the recovery is essential, but merge barriers (a 16-line tokenizer configuration change) eliminate the need for it entirely. An additional 8% of heads collapse into position-zero sinks that contribute nothing (confirmed by ablation: +1.4% change when removed). Together, over half of attention capacity in standard BPE is non-productive. We establish these findings through a developmental atlas: the first comprehensive tracking of attention head specialization at realistic scale, covering 384 headsacross 7 behavior types, 131 checkpoints per run, and six training runs on two corpora (GPT-NeoX 410M). Key results: (1) spacing is a fixed capacity tax of ~45% of heads regardless of corpus composition, confirmed by a falsified prediction on a mixed corpus (2) P0 sinking is a failure cascade at 8.3%, with the first causal proof that P0 heads contribute nothing (3) BPE damage operates in two regimes (spacing on web text, stranding on structured data) that form a continuum (4) merge barriers are universal across character sets. Third paper in a research program on tokenizer-attention coupling (DOI: 10.5281/zenodo.20925910) and stranded attention (DOI: 10.5281/zenodo.21158886).

Citations

• Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
• What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
• Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
• The emergence of sparse attention: impact of data distribution and benefits of repetition
• Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
• Language Models Grow Less Humanlike beyond Phase Transition
• Which Attention Heads Matter for In-Context Learning?
• One-layer transformers fail to solve the induction heads task
• Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation
• The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains
• OLMo: Accelerating the Science of Language Models
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
• The mechanistic basis of data dependence and abrupt learning in an in-context classification task
• Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
• Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
• Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability
• Birth of a Transformer: A Memory Viewpoint
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?
• In-context Learning and Induction Heads
• Data Distributional Properties Drive Emergent In-Context Learning in Transformers
• An Explanation of In-context Learning as Implicit Bayesian Inference
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Scaling Laws for Neural Language Models
• Unsupervised Cross-lingual Representation Learning at Scale
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• An Empirical Model of Large-Batch Training

Cited by

• The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models

Related

• Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
• Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
• Low-Rank Bottleneck in Multi-head Attention Models
• GTA: Grouped-head latenT Attention
• Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models