Improving Transformers using Faithful Positional Encoding

2024/05/15 by Tsuyoshi Idé, Idé, Tsuyoshi, Jokin Labaien +3
Engineering · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.2405.09061

openalex publication_date 2024/05/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.

Related

• Alternative positional encoding functions for neural transformers
• Positional Encoding in Transformer-Based Time Series Models: A Survey
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
• Relative Positional Encoding for Transformers with Linear Complexity