Formal Algorithms for Transformers

2022/07/19 by Mary Phuong, Marcus Hutter · 3 voices · 9 citations
#cs.LG #cs.AI #cs.CL #cs.NE

paper · pdf

Abstract

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

Cited by

• Topological constraints on self-organization in locally interacting systems
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Generative Bayesian Optimization: Generative Models as Acquisition Functions
• Amortized Active Generation of Pareto Sets
• MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems
• What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture
• InTraVisTo: Inside Transformer Visualisation Tool
• Normalization (machine learning) [wikipedia]
• Transformer (deep learning) [wikipedia]

Discussions

• Formal Algorithms for Transformers [hn, 106 points, 13 comments]
• It's not only the sloppy review process but a lot of AI papers just contain prose and vague descriptions. One might argue that acceptance is based on in-group background knowledge and I feel that's of [bsky, 4 points, 1 comments]
• it's so useful that Shalizi has pulled all these actually readable papers on transformers because, as he says, "...the sheer opacity of this literature is I think a real problem" arxiv.org/pdf/2207.09 [bsky, 3 points, 2 comments]

Related

• An Introduction to Transformers
• Introduction to Transformers: an NLP Perspective
• Thinking Like Transformers
• Transformer models: an introduction and catalog
• The Transformer Cookbook