Probabilistic Transformers

2020/10/15 by Javier R. Movellan, Movellan, Javier R., Prasad Gabbur +1
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2010.15583

openalex publication_date 2020/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that Transformers are Maximum Posterior Probability estimators for Mixtures of Gaussian Models. This brings a probabilistic point of view to Transformers and suggests extensions to other probabilistic cases.

Citations

• Neural Machine Translation by Jointly Learning to Align and Translate
• Hopfield Networks is All You Need
• Attention that does not Explain Away

Related

• What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture
• Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixtures
• Pathologies in priors and inference for Bayesian transformers
• Maximum Likelihood Estimates for Gaussian Mixtures Are Transcendental
• Transformers can optimally learn regression mixture models