The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training

2025/02/15 by Matteo Saponati, Pascal Sager, Saponati, Matteo +8 · 1 voice · 2 citations
Computer Science · Engineering · Mathematics · Psychology · #Cognitive Science and Mapping #Cognitive psychology #Directionality #Electrical engineering #Engineering #Geometry #Mathematics #Psychology #Symmetry (geometry) #Transformer

paper · pdf · doi:10.48550/arxiv.2502.10927

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models - including ModernBERT, GPT, LLaMA3, and Mistral - and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models.

Cited by

• Reversal Invariance in Autoregressive Language Models
• GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts

Discussions

• 1/ I am very excited to announce that our paper "The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training" is available on arXiv 💜 arxiv.o [bsky, 8 points, 1 comments]

Related

• Guiding Attention for Self-Supervised Learning with Transformers
• On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
• Revisiting Transformers with Insights from Image Filtering and Boosting