Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

2024/03/28 by Samuel Marks, Can Rager, Eric J. Michaud +3 · 1 voice · 17 citations
Computer Science · #cs.LG #cs.AI #cs.CL

paper · pdf

arxiv published 2024/03/28 · arxiv updated 2025/03/27

Abstract

We introduce methods for discovering and applying sparse feature circuits. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.

Cited by

• Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
• LLMs Can Annotate Attribution Graphs
• Fast Multi-dimensional Refusal Subspaces via RFM-AGOP
• Scalable Circuit Learning for Interpreting Large Language Models
• MonoLoss: A Training Objective for Interpretable Monosemantic Representations
• Data-driven Circuit Discovery for Interpretability of Language Models
• Features have life history. And we should care
• Retrieval and competition: how a protein foundation model starts a protein
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
• Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition
• Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
• Scaling sparse feature circuit finding for in-context learning
• MIB: A Mechanistic Interpretability Benchmark
• CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
• Mood Matters: How Syntactic Sensitivity Undermines Safety Alignment
• The Ignition Index: Measuring Global Workspace Dynamics in Language Models
• Layer-wise Positional Bias in Short-Context Language Modeling

Discussions

• (ICLR) Sparse feature circuits give us a way to understand and 𝗲𝗱𝗶𝘁 how an LLM performs a given task. We don't need to hypothesize what this algorithm is ahead of time—a huge advantage over other [bsky, 1 points, 1 comments]

Related

• Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models
• Finding Neurons in a Haystack: Case Studies with Sparse Probing
• Weight-sparse transformers have interpretable circuits
• Scalable Circuit Learning for Interpreting Large Language Models
• Scaling sparse feature circuit finding for in-context learning