Normalization in Attention Dynamics

2025/10/24 by Karagodin, Nikita, Ge, Shu, Polyanskiy, Yury +1 · 2 citations
#35Q68 #37C10 #37N99 #68T07 #82C22 #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.22026

Abstract

We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice.

Citations

• A multiscale analysis of mean-field transformers in the moderate interaction regime
• Quantitative Clustering in Mean-Field Transformer Models
• The Curse of Depth in Large Language Models
• Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
• Measure-to-measure interpolation using Transformers
• Clustering in Causal Attention Masking
• Emergence of meta-stable clustering in mean-field transformer models
• Dynamic metastability in the self-attention model
• nGPT: Normalized Transformer with Representation Learning on the Hypersphere
• A deeper look at depth pruning of LLMs
• Compact Language Models via Pruning and Knowledge Distillation
• Synchronization on circles and spheres with nonlinear interactions
• The Unreasonable Ineffectiveness of the Deeper Layers
• Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models
• Geometric Dynamics of Signal Propagation Predict Trainability of Transformers
• A mathematical perspective on Transformers
• Small-scale proxies for large-scale Transformer training instabilities
• The emergence of clusters in self-attention dynamics
• LLaMA: Open and Efficient Foundation Language Models
• Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
• Sinkformers: Transformers with Doubly Stochastic Attention
• Root Mean Square Layer Normalization
• Attention Is All You Need
• Identity Mappings in Deep Residual Networks
• Neural Machine Translation by Jointly Learning to Align and Translate

Cited by

• Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

Related

• Limitations of Normalization in Attention Mechanism
• The Mean-Field Dynamics of Transformers
• Normalized Attention Without Probability Cage
• Benign Overfitting in Token Selection of Attention Mechanism
• Incorporating Residual and Normalization Layers into Analysis of Masked Language Models