Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition

2025/04/29 by Z. T. He, Zhengfu He, He, Zhengfu +14 · 1 voice · 2 citations
Computer Science · Engineering · Materials Science · #Advanced Neural Network Applications #Associative property #Autoencoder #Computation and Language (cs.CL) #FOS: Computer and information sciences #Interpretability #Low-power high-performance VLSI design #Machine Learning (cs.LG) #Machine Learning in Materials Science #Neural coding #Resampling #Sparse matrix #Transformer #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2504.20938

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/04/29 · arxiv published 2025/04/29 · arxiv updated 2025/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions. We show that Lorsa heads find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads and attention sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse Autoencoder (SAE) are both sparse dictionary learning methods applied to different Transformer components, and lead to consistent findings in many ways. For instance, we discover a comprehensive family of arithmetic-specific Lorsa heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads. We also conduct extensive experiments on architectural design ablation, Lorsa scaling law and error analysis.

Citations

• Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
• Decomposing The Dark Matter of Sparse Autoencoders
• Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures
• An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
• Interpreting Attention Layer Outputs with Sparse Autoencoders
• Transcoders Find Interpretable LLM Feature Circuits
• Scaling and evaluating sparse autoencoders
• Automatically Identifying Local and Global Circuits with Linear Computation Graphs
• Retrieval Head Mechanistically Explains Long-Context Factuality
• Improving Dictionary Learning with Gated Sparse Autoencoders
• Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT
• Successor Heads: Recurring, Interpretable Attention Heads In The Wild
• Copy Suppression: Comprehensively Understanding an Attention Head
• Efficient Streaming Language Models with Attention Sinks
• Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Finding Neurons in a Haystack: Case Studies with Sparse Probing
• How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
• Towards Automated Circuit Discovery for Mechanistic Interpretability
• A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
• Toy Models of Superposition
• Locating and Editing Factual Associations in GPT
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Cited by

• Adversarial Attacks Leverage Interference Between Features in Superposition
• Attention Layers Add Into Low-Dimensional Residual Subspaces

Discussions

• [2504.20938] Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition arxiv.org/abs/2504.209... [bsky, 1 points, 0 comments]

Related

• Intrinsically Interpretable Attention via Sparse Post-Training
• Interpreting Attention Layer Outputs with Sparse Autoencoders
• Attention Layers Add Into Low-Dimensional Residual Subspaces
• Low-Rank Bottleneck in Multi-head Attention Models
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth