Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability

2023/05/04 by Ziming Liu, Liu, Ziming, Eric Gan +3 · 4 citations
Computer Science · #Explainable Artificial Intelligence (XAI) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2305.08746

Abstract

We introduce Brain-Inspired Modular Training (BIMT), a method for making neural networks more modular and interpretable. Inspired by brains, BIMT embeds neurons in a geometric space and augments the loss function with a cost proportional to the length of each neuron connection. We demonstrate that BIMT discovers useful modular neural networks for many simple tasks, revealing compositional structures in symbolic formulas, interpretable decision boundaries and features for classification, and mathematical structure in algorithmic datasets. The ability to directly see modules with the naked eye can complement current mechanistic interpretability strategies such as probes, interventions or staring at all weights.

Cited by

• Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
• Interpretable Machine Learning for Life Expectancy Prediction: A Comparative Study of Linear Regression, Decision Tree, and Random Forest
• A Bio-Inspired Minimal Model for Non-Stationary K-Armed Bandits
• Pruning Increases Orderedness in Recurrent Computation

Related

• Evaluating Brain-Inspired Modular Training in Automated Circuit Discovery for Mechanistic Interpretability
• Modular Networks: Learning to Decompose Neural Computation
• Training Neural Networks for Modularity aids Interpretability
• The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms
• When and why modular representations emerge