Critical attention scaling in long-context transformers

2025/10/07 by Shi Chen, Zhengjiang Lin, Chen, Shi +5 · 2 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Classical Analysis and ODEs (math.CA) #Discrete Mathematics (cs.DM) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #cs.AI #cs.DM #cs.LG #math.CA

paper · pdf · doi:10.48550/arxiv.2510.05554

published as Proceedings of the Fourteenth International Conference on Learning Representations (ICLR 2026), 2026 · 31 pages, 2 figures

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length n increases, causing tokens to cluster excessively, a phenomenon known as rank-collapse. While attention scaling effectively addresses this deficiency by rescaling attention scores with a polylogarithmic factor β_n, theoretical justification for this approach remains lacking. We analyze a simplified yet tractable model that magnifies the effect of attention scaling. In this model, attention exhibits a phase transition governed by the scaling factor β_n: insufficient scaling collapses all tokens to a single direction, while excessive scaling reduces attention to identity, thereby eliminating meaningful interactions between tokens. Our main result identifies the critical scaling β_n \asymp log n and provides a rigorous justification for attention scaling in YaRN and Qwen, clarifying why logarithmic scaling maintains sparse, content-adaptive attention at large context lengths.

Citations

• The Mean-Field Dynamics of Transformers
• A multiscale analysis of mean-field transformers in the moderate interaction regime
• Synchronization of mean-field models on the circle
• Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
• Quantitative Clustering in Mean-Field Transformer Models
• SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
• Scalable-Softmax Is Superior for Attention
• The random graph process is globally synchronizing
• Residual connections provably mitigate oversmoothing in graph neural networks
• Clustering in Causal Attention Masking
• Emergence of meta-stable clustering in mean-field transformer models
• Dynamic metastability in the self-attention model
• Geometric Dynamics of Signal Propagation Predict Trainability of Transformers
• A mathematical perspective on Transformers
• YaRN: Efficient Context Window Extension of Large Language Models
• The emergence of clusters in self-attention dynamics
• Expander graphs are globally synchronizing
• Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Longformer: The Long-Document Transformer
• Deep Residual Learning for Image Recognition
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Cited by

• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
• A multiscale analysis of mean-field transformers in the moderate interaction regime

Related

• Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
• Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processing
• Scale-invariant Attention
• Curse of High Dimensionality Issue in Transformer for Long-context Modeling
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration