2024/06/11 by Boyang Li, Yulin Wu, Li, Boyang +4 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #Algorithms and Data Compression #DNA and Biological Computing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Modular Robots and Swarm Intelligence #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.2406.07049
openalex publication_date 2024/06/11 · openalex created_date 2024/06/13 · openalex updated_date 2026/07/28
Understanding spatial relationships across all dimensions is fundamental for intelligent systems. However, existing positional embeddings, such as Rotary Positional Embedding (RoPE), lack theoretical guarantees for high-dimensional spatiotemporal tasks like video understanding and robotic navigation. Inspired by the hexagonal periodic coding of grid cells in mammalian spatial cognition, we propose GridPE -- a novel positional embedding framework that integrates computational neuroscience principles with harmonic analysis. Our approach builds upon Random Fourier Features and leverages principles from neuroscience to construct efficient embeddings. Theoretically, we prove that any translation-invariant spatial function can be approximated by a finite sum of Fourier bases, which naturally reduces to RoPE in the one-dimensional case. We then derive the directions and quantities of frequency vectors at each scale in any Euclidean dimension, along with the optimal ratio between different scales, from a bioavailability perspective. These derivations are equivalent to the relationship between the centroid and the vertices of a regular simplex in that dimension. We validate GridPE across a range of spatial modeling tasks, including 2D image classification (ImageNet100) and 3D point cloud recognition (ModelNet40). Our theoretical analysis establishes GridPE as a unified framework for positional embedding in arbitrary-dimensional spaces, while empirical results demonstrate its superiority over existing methods.