Validating Mechanistic Interpretations: An Axiomatic Approach

2024/07/18 by Nils Palumbo, Ravi Mangal, Palumbo, Nils +9
Computer Science · #FOS: Computer and information sciences #Formal Methods in Verification #Machine Learning (cs.LG) #Model-Driven Software Engineering Techniques

paper · pdf · doi:10.48550/arxiv.2407.13594

openalex publication_date 2024/07/18 · openalex created_date 2025/01/04 · openalex updated_date 2026/07/28

Abstract

Mechanistic interpretability aims to reverse engineer the computation performed by a neural network in terms of its internal components. Although there is a growing body of research on mechanistic interpretation of neural networks, the notion of a mechanistic interpretation itself is often ad-hoc. Inspired by the notion of abstract interpretation from the program analysis literature that aims to develop approximate semantics for programs, we give a set of axioms that formally characterize a mechanistic interpretation as a description that approximately captures the semantics of the neural network under analysis in a compositional manner. We demonstrate the applicability of these axioms for validating mechanistic interpretations on an existing, well-known interpretability study as well as on a new case study involving a Transformer-based model trained to solve the well-known 2-SAT problem.

Related

• Open Problems in Mechanistic Interpretability
• A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i
• A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models
• Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
• Combining Causal Models for More Accurate Abstractions of Neural Networks