Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

2024/03/26 by Michael Hanna, Sandro Pezzelle, Hanna, Michael +3 · 26 citations
Business, Management and Accounting · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Islamic Finance and Banking Studies #Machine Learning (cs.LG) #Religion, Society, and Development

paper · pdf · doi:10.48550/arxiv.2403.17806

openalex publication_date 2024/03/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many recent language model (LM) interpretability studies have adopted the circuits framework, which aims to find the minimal computational subgraph, or circuit, that explains LM behavior on a given task. Most studies determine which edges belong in a LM's circuit by performing causal interventions on each edge independently, but this scales poorly with model size. Edge attribution patching (EAP), gradient-based approximation to interventions, has emerged as a scalable but imperfect solution to this problem. In this paper, we introduce a new method - EAP with integrated gradients (EAP-IG) - that aims to better maintain a core property of circuits: faithfulness. A circuit is faithful if all model edges outside the circuit can be ablated without changing the model's performance on the task; faithfulness is what justifies studying circuits, rather than the full model. Our experiments demonstrate that circuits found using EAP are less faithful than those found using EAP-IG, even though both have high node overlap with circuits found previously using causal interventions. We conclude more generally that when using circuits to compare the mechanisms models use to solve tasks, faithfulness, not overlap, is what should be measured.

Cited by

• Mechanistic Analysis of Circuit Preservation in Federated Learning
• A Mechanistic Perspective and Difficulty Metric for Unlearning
• Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?
• A Geometric Unification of Concept Learning with Concept Cones
• Emergence and Localisation of Semantic Role Circuits in LLMs
• Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
• Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
• Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models
• APP: Accelerated Path Patching with Task-Specific Pruning
• Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods
• BlackboxNLP-2025 MIB Shared Task: Improving Circuit Faithfulness via Better Edge Selection
• PAHQ: Accelerating Automated Circuit Discovery through Mixed-Precision Inference Optimization
• The Curious Case of Factual (Mis)Alignment between LLMs' Short- and Long-Form Answers
• Discursive Circuits: How Do Language Models Understand Discourse Relations?
• BlackboxNLP-2025 MIB Shared Task: Exploring Ensemble Strategies for Circuit Localization Methods
• PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHing
• Mechanistic Interpretability as Statistical Estimation: A Variance Analysis
• ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attack
• Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
• Query Circuits: Explaining How Language Models Answer User Prompts
• Discovering Transformer Circuits via a Hybrid Attribution and Pruning Framework
• Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer
• Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs
• Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models
• Evolution of Concepts in Language Model Pre-Training
• Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining

Related

• RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching
• EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification
• Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis
• Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models
• Attribution Patching Outperforms Automated Circuit Discovery