Increasing transformer token length with a Maximum Entropy Principle Method

2024/08/17 by Robert I. Cukier, Cukier, R. I.
Engineering · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Magnetic confinement fusion research #Particle accelerators and beam dynamics

paper · pdf · doi:10.48550/arxiv.2408.10277

openalex publication_date 2024/08/17 · openalex created_date 2024/10/01 · openalex updated_date 2026/07/28

Abstract

Transformers suffer from the computational overhead of their quadratic dependence on the length of sequences processed. We present three methods, all adding an intermediate step between training and inference/generation, which extend the autoregressive length of transformers. All rely on a Maximum Entropy Principle (MEP) whereby entropy is maximized in the presence of suitable constraints, accounted for by use of Lagrange Multipliers. These constraint methods extend the autoregressive character from T to 2T tokens in a linear-with-T fashion. There is overhead associated with this added step, but they should still be faster than the standard methods.

Related

• Quantitative Bounds for Length Generalization in Transformers
• Deriving the Scaled-Dot-Function via Maximum Likelihood Estimation and Maximum Entropy Approach
• Transformer Quality in Linear Time
• Transformers Can Achieve Length Generalization But Not Robustly
• Uncovering mesa-optimization algorithms in Transformers