The Transformer Cookbook

2025/10/01 by Andy Yang, Yang, Andy, Christopher Watson +15 · 1 voice · 1 citation
#cs.LG

paper · pdf · doi:10.48550/arxiv.2510.00368

Abstract

We present the transformer cookbook: a collection of techniques for directly encoding algorithms into a transformer's parameters. This work addresses the steep learning curve of such endeavors, a problem exacerbated by a fragmented literature where key results are scattered across numerous papers. In particular, we synthesize this disparate body of findings into a curated set of recipes that demonstrate how to implement everything from basic arithmetic in feed-forward layers to complex data routing via self-attention. Our mise en place of formulations is for both newcomers seeking an accessible entry point and experts in need of a systematic reference. This unified presentation of transformer constructions provides a foundation for future work spanning theoretical research in computational complexity to empirical investigations in architecture design and interpretability.

Citations

• A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
• Simulating Hard Attention Using Soft Attention
• One-layer transformers fail to solve the induction heads task
• Separations in the Representational Capabilities of Transformers and Recurrent Architectures
• Understanding Transformer Reasoning Capabilities via Graph Algorithms
• Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers
• Transformers as Transducers
• Why are Sensitive Functions Hard for Transformers?
• Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages
• Logical Languages Accepted by Transformer Encoders with Hard Attention
• Representational Strengths and Limitations of Transformers
• The Impact of Positional Encoding on Length Generalization in Transformers
• Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
• On the Expressivity Role of LayerNorm in Transformers' Attention
• Tighter Bounds on the Expressivity of Transformer Encoders
• Tracr: Compiled Transformers as a Laboratory for Interpretability
• What learning algorithm is in-context learning? Investigations with linear models
• The Parallelism Tradeoff: Limitations of Log-Precision Transformers
• Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity
• Overcoming a Theoretical Limitation of Self-Attention
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Saturated Transformers are Constant-Depth Threshold Circuits
• Thinking Like Transformers
• Self-Attention Networks Can Process Bounded Hierarchical Languages
• Transformer Feed-Forward Layers Are Key-Value Memories
• Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
• Self-Attention with Relative Position Representations
• Spectrally-normalized margin bounds for neural networks
• Attention Is All You Need
• Layer Normalization
• Gaussian Error Linear Units (GELUs)
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Neural Machine Translation by Jointly Learning to Align and Translate
• Multilayer feedforward networks are universal approximators
• Cognitron: A self-organizing multilayered neural network
• The Expressive Power of Transformers with Chain of Thought
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Why Are Linear RNNs More Parallelizable?

Discussions

• We present The Transformer Cookbook: a collection of recipes for programming algorithms directly into transformers! Hungry for an induction head? Craving a Dyck language recognizer? We show you step-b [bsky, 5 points, 1 comments]

Related

• Formal Algorithms for Transformers
• An Introduction to Transformers
• Learning Transformer Programs
• Looped Transformers as Programmable Computers
• Thinking Like Transformers