Transformers are Universal Predictors

2023/07/15 by Basu, Sourya, Choraria, Moulik, Varshney, Lav R.
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2307.07843

Abstract

We find limits to the Transformer architecture for language modeling and show it has a universal prediction property in an information-theoretic sense. We further analyze performance in non-asymptotic data regimes to understand the role of various components of the Transformer architecture, especially in the context of data-efficient training. We validate our theoretical analysis with experiments on both synthetic and real datasets.

Related

• Understanding Transformers via N-gram Statistics
• Can Transformers Learn n-gram Language Models?
• On the Universality of Transformer Architectures; How Much Attention Is Enough?
• Transformers Can Represent n-gram Language Models
• Introduction to Transformers: an NLP Perspective