The Transformer as a Polar State Estimator

2026/05/31 by Peter Racioppo
Computer Science · #cs.LG #cs.AI

paper · pdf

arxiv created 2026/08/02 · arxiv updated 2026/08/04

Abstract

We show that the core components of the Transformer---attention, residual connections, and normalization---arise naturally from a single geometric state estimation problem. Modeling the latent state in polar coordinates naturally separates radial and hyperspherical dynamics, yielding a precision-weighted filtering procedure in which normalization enforces the hyperspherical constraint, attention aggregates directional evidence, and the residual connection implements an incremental state update. The standard Transformer block with rotary positional encodings is recovered by discarding the geometric correction terms of the resulting state estimator, showing that its architecture follows from the underlying estimation problem rather than from independent design choices. The proposed Polar Transformer retains these geometric corrections.

Citations

• Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
• RiemannFormer: A Framework for Attention in Curved Spaces
• Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
• PaTH Attention: Position Encoding via Accumulating Householder Transformations
• nGPT: Normalized Transformer with Representation Learning on the Hypersphere
• LieRE: Lie Rotational Positional Encodings
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• A mathematical perspective on Transformers
• Can a Transformer Represent a Kalman Filter?
• Traveling Words: A Geometric Interpretation of Transformers
• Retentive Network: A Successor to Transformer for Large Language Models
• A Length-Extrapolatable Transformer
• Efficiently Modeling Long Sequences with Structured State Spaces
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Probabilistic Attention for Interactive Segmentation
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Kalman Filters on Differentiable Manifolds
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Hopfield Networks is All You Need
• Transformer Dissection: A Unified Understanding of Transformer's\n Attention via the Lens of Kernel
• Attention Is All You Need
• The invariant extended Kalman filter as a stable observer