Position Information in Transformers: An Overview

2021/02/22 by Philipp Dufter, Dufter, Philipp, Martin Schmitt +3 · 19 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2102.11090

First two authors contributed equally

openalex publication_date 2021/02/22 · arxiv created 2021/09/09 · arxiv updated 2021/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformers are arguably the main workhorse in recent Natural Language Processing research. By definition a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; (4) provide stimuli for future research.

Cited by

• LithoFormer: A Robust Framework for Stratigraphic Inference via Transformers
• Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs
• Alternative positional encoding functions for neural transformers
• Softmax Transformers are Turing-Complete
• Weaver: Kronecker Product Approximations of Spatiotemporal Attention for Traffic Network Forecasting
• On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility
• LayerNorm Induces Recency Bias in Transformer Decoders
• Explicit Context-Driven Neural Acoustic Modeling for High-Fidelity RIR Generation
• Collaborative local–global context modeling for session-based recommendation
• What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture
• What do language models model? Transformers, automata, and the format of thought
• Fractal Language Modelling by Universal Sequence Maps (USM)
• Enhancing Temporal Sensitivity of Large Language Model for Recommendation with Counterfactual Tuning
• CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
• Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability
• Equivariant Spherical Transformer for Efficient Molecular Modeling
• Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
• PaTH Attention: Position Encoding via Accumulating Householder Transformations
• The effect of the number of parameters and the number of local feature patches on loss landscapes in distributed quantum neural networks
• Distilling Specialized Orders for Visual Generation

Related

• Position Information in Transformers: An Overview
• Multiplicative Position-aware Transformer Models for Language Understanding
• Improve Transformer Models with Better Relative Position Embeddings
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• A Simple and Effective Positional Encoding for Transformers