Arbitrary-Length Generalization for Addition in a Tiny Transformer

2024/05/31 by Alexandre G. Patriota, Patriota, Alexandre Galvao
Computer Science · Engineering · Mathematics · #Advanced Numerical Analysis Techniques #Advanced Optimization Algorithms Research #Applications (stat.AP) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Matrix Theory and Algorithms

paper · pdf · doi:10.48550/arxiv.2406.00075

openalex publication_date 2024/05/31 · openalex created_date 2024/06/06 · openalex updated_date 2026/07/28

Abstract

This paper introduces a novel training methodology that enables a Transformer model to generalize the addition of two-digit numbers to numbers with unseen lengths of digits. The proposed approach employs an autoregressive generation technique, processing from right to left, which mimics a common manual method for adding large numbers. To the best of my knowledge, this methodology has not been previously explored in the literature. All results are reproducible, and the corresponding R code is available at github.com/AGPatriota/ALGA-R/.

Related

• Length Generalization in Arithmetic Transformers
• Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
• Looped Transformers for Length Generalization
• Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure
• Understanding Addition in Transformers