Decomposition of Small Transformer Models

2025/11/12 by Casper L. Christensen, Christensen, Casper L., Logan Riggs +1
Computer Science · Physics and Astronomy · #Decomposition #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Function (biology) #Generative Adversarial Networks and Image Synthesis #Interpretability #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Parameter space #Range (aeronautics) #Space (punctuation) #Transformer

paper · pdf · doi:10.48550/arxiv.2511.08854

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/11/12 · openalex created_date 2025/11/14 · openalex updated_date 2026/07/28

Abstract

Recent work in mechanistic interpretability has shown that decomposing models in parameter space may yield clean handles for analysis and intervention. Previous methods have demonstrated successful applications on a wide range of toy models, but the gap to "real models" has not yet been bridged. In this work, we extend Stochastic Parameter Decomposition (SPD) to Transformer models, proposing an updated causal importance function suited for sequential data and a new loss function. We demonstrate that SPD can successfully decompose a toy induction-head model and recover the expected 2-step circuit. We also show that applying SPD to GPT-2-small can successfully locate subcomponents corresponding to interpretable concepts like "golf" and "basketball". These results take the first step in the direction of extending SPD to modern models, and show that we can use the method to surface interpretable parameter-space mechanisms.

Citations

• Stochastic Parameter Decomposition
• Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition
• Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
• Flex Attention: A Programming Model for Generating Optimized Attention Kernels
• A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
• In-context Learning and Induction Heads
• Toy Models of Superposition
• Locating and Editing Factual Associations in GPT
• Explaining by Removing: A Unified Framework for Model Explanation
• Attention Is All You Need
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks

Related

• Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains
• Allocation of Parameters in Transformers
• Small Singular Values Matter: A Random Matrix Analysis of Transformer Models
• What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture
• Analyzing Transformers in Embedding Space