Algebraic Positional Encodings

2023/12/26 by Konstantinos Kogkalidis, Kogkalidis, Konstantinos, Jean-Philippe Bernardy +3 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.16045

openalex publication_date 2023/12/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a novel positional encoding strategy for Transformer-style models, addressing the shortcomings of existing, often ad hoc, approaches. Our framework provides a flexible mapping from the algebraic specification of a domain to an interpretation as orthogonal operators. This design preserves the algebraic characteristics of the source domain, ensuring that the model upholds its desired structural properties. Our scheme can accommodate various structures, ncluding sequences, grids and trees, as well as their compositions. We conduct a series of experiments to demonstrate the practical applicability of our approach. Results suggest performance on par with or surpassing the current state-of-the-art, without hyper-parameter optimizations or "task search" of any kind. Code is available at https://github.com/konstantinosKokos/ape.

Cited by

• Tree-Based Deep Learning for Ranking Symbolic Integration Algorithms
• Positional Encoding meets Persistent Homology on Graphs

Related

• CoPE: A Lightweight Complex Positional Encoding
• Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Improve Transformer Models with Better Relative Position Embeddings
• ExPe: Exact Positional Encodings for Generative Transformer Models with Extrapolating Capabilities