Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding

2021/06/05 by Yang Li, Si Si, Li, Yang +7 · 15 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2106.02795

openalex publication_date 2021/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e.g., pixel positions on an image, where L₂ distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.

Citations

• Backpropagation Applied to Handwritten Zip Code Recognition
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network
• Reformer: The Efficient Transformer
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Cited by

• RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
• KeyPointDiffuser: Unsupervised 3D Keypoint Learning via Latent Diffusion Models
• VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation
• Leveraging an Atmospheric Foundational Model for Subregional Sea Surface Temperature Forecasting
• ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
• An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
• LightGlueStick: a Fast and Robust Glue for Joint Point-Line Matching
• Referring Expression Comprehension for Small Objects
• Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation
• What Happens Next? Anticipating Future Motion by Generating Point Trajectories
• Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems
• Enhancing Knowledge Tracing through Leakage-Free and Recency-Aware Embeddings
• Improving Greenland Bed Topography Mapping with Uncertainty-Aware Graph Learning on Sparse Radar Data
• Anisotropic Fourier Features for Positional Encoding in Medical Imaging
• Score-informed Neural Operator for Enhancing Ordering-based Causal Discovery

Related

• Rethinking Positional Encoding
• Local Positional Encoding for Multi-Layer Perceptrons
• Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding
• Trading Positional Complexity vs. Deepness in Coordinate Networks
• A 2D Semantic-Aware Position Encoding for Vision Transformers