Space filling positionality and the Spiroformer

2025/07/11 by Maurin, M., Evangelista-Alvarado, M. Á., Suárez-Serrato, P.
#Artificial Intelligence (cs.AI) #Differential Geometry (math.DG) #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Symplectic Geometry (math.SG)

paper · doi:10.48550/arxiv.2507.08456

Abstract

Transformers excel when dealing with sequential data. Generalizing transformer models to geometric domains, such as manifolds, we encounter the problem of not having a well-defined global order. We propose a solution with attention heads following a space-filling curve. As a first experimental example, we present the Spiroformer, a transformer that follows a polar spiral on the 2-sphere.

Citations

• SymFlux: deep symbolic regression of Hamiltonian vector fields
• A Structure-Preserving Kernel Method for Learning Hamiltonian Systems
• Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Networks
• Spherical Transformer for LiDAR-based 3D Recognition
• Symmetry Group Equivariant Architectures for Physics
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Geomstats: A Python Package for Riemannian Geometry in Machine Learning
• Spherical CNNs
• Attention Is All You Need
• SO(3)-Equivariant Neural Networks for Learning from Scalar and Vector Fields on Spheres

Related

• RiemannFormer: A Framework for Attention in Curved Spaces
• Spherical Position Encoding for Transformers
• The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
• Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
• Spacetime E(n)-Transformer: Equivariant Attention for Spatio-temporal Graphs