Evaluating Attribution Methods using White-Box LSTMs

2020/10/16 by Yiding Hao, Hao, Yiding
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2010.08606

openalex publication_date 2020/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Interpretability methods for neural networks are difficult to evaluate because we do not understand the black-box models typically used to test them. This paper proposes a framework in which interpretability methods are evaluated using manually constructed networks, which we call white-box networks, whose behavior is understood a priori. We evaluate five methods for producing attribution heatmaps by applying them to white-box LSTM classifiers for tasks based on formal languages. Although our white-box classifiers solve their tasks perfectly and transparently, we find that all five attribution methods fail to produce the expected model explanations.

Citations

• Finding Structure in Time
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Axiomatic Attribution for Deep Networks
• Methods for Interpreting and Understanding Deep Neural Networks
• Sanity Checks for Saliency Maps
• Understanding Recurrent Neural Architectures by Analyzing and Synthesizing Long Distance Dependencies in Benchmark Sequential Datasets
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Benchmarking Attribution Methods with Relative Feature Importance

Related

• Time to Focus: A Comprehensive Benchmark Using Time Series Attribution Methods
• AttributionLab: Faithfulness of Feature Attribution Under Controllable Environments
• MFABA: A More Faithful and Accelerated Boundary-based Attribution Method for Deep Neural Networks
• CHALLENGER: Training with Attribution Maps
• Attribution Explanations for Deep Neural Networks: A Theoretical Perspective