Self-Attention as Distributional Projection: A Unified Interpretation of Transformer Architecture

2025/11/16 by Mehta, Nihal
Neuroscience · Psychology · Engineering · #Embodied and Extended Cognition #Philosophy and Theoretical Science #Ferroelectric and Negative Capacitance Devices

paper · doi:10.48550/arxiv.2511.13780

Abstract

This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context. Starting from the co-occurrence matrix underlying GloVe embeddings, we demonstrate how the projection naturally captures contextual influence, with the query-key-value mechanism arising as the natural asymmetric extension for modeling directional relationships. Positional encodings and multi-head attention then follow as structured refinements of this same projection principle. Our analysis demonstrates that the Transformer architecture's particular algebraic form follows from these projection principles rather than being an arbitrary design choice.

Related

• The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
• On Identifiability in Transformers
• Generalized Attention Mechanism and Relative Position for Transformer
• Quantifying Attention Flow in Transformers
• Telling BERT's full story: from Local Attention to Global Aggregation