What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

2026/03/31 by Runyu Peng, Ruixiao Li, Mingshu Chen +5
Computer Science · #cs.LG #cs.CL

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/04

Abstract

Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks. Causal large language models reliably form one at position zero, though its role remains debated. We approach this question from a mechanistic perspective, tracing how the position-zero sink arises from the model's internal computation. We identify a two-block subnetwork responsible for this behavior, which we term the P0-Sink Circuit, and show it arises purely from the structural properties of causal attention, requiring no semantic content. We further validate through from-scratch pre-training experiments that two proposed parameter-free methods effectively accelerate P0 sink formation, and find that earlier sink formation benefits pre-training and improves downstream performance. Both methods outperform the Transformer baseline and achieve performance comparable to Gated Attention across comprehensive settings. Code is available now at https://github.com/Pryest/flash-linear-attention.

Citations

• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
• Vision Transformers Don't Need Trained Registers
• The emergence of sparse attention: impact of data distribution and benefits of repetition
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
• Identifying and Evaluating Inactive Heads in Pretrained LLMs
• Why do LLMs attend to the first token?
• Systematic Outliers in Large Language Models
• Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
• Rope to Nope and Back Again: A New Hybrid Attention Strategy
• When Attention Sink Emerges in Language Models: An Empirical View
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration
• Massive Activations in Large Language Models
• Efficient Streaming Language Models with Attention Sinks
• LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
• Theoretical Foundations of t-SNE for Visualizing High-Dimensional Clustered Data
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Talking-Heads Attention
• Root Mean Square Layer Normalization
• Pointer Sentinel Mixture Models
• Layer Normalization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature