Where do Models go Wrong? Parameter-Space Saliency Maps for\n Explainability

2021/08/03 by R. V. Levin, Roman Levin, Manli Shu +10 · 1 voice · 1 citation
Computer Science · #Explainable Artificial Intelligence (XAI) #Adversarial Robustness in Machine Learning #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2108.01335

Abstract

Conventional saliency maps highlight input features to which neural network\npredictions are highly sensitive. We take a different approach to saliency, in\nwhich we identify and analyze the network parameters, rather than inputs, which\nare responsible for erroneous decisions. We find that samples which cause\nsimilar parameters to malfunction are semantically similar. We also show that\npruning the most salient parameters for a wrongly classified sample often\nimproves model behavior. Furthermore, fine-tuning a small number of the most\nsalient parameters on a single sample results in error correction on other\nsamples that are misclassified for similar reasons. Based on our parameter\nsaliency method, we also introduce an input-space saliency technique that\nreveals how image features cause specific network components to malfunction.\nFurther, we rigorously validate the meaningfulness of our saliency maps on both\nthe dataset and case-study levels.\n

Cited by

• Identification of Attack-Specific Signatures in Adversarial Examples

Discussions

• Where Do Models Go Wrong? Parameter-Space Saliency Maps for Explainability [hn, 84 points, 6 comments]

Related

• Investigating sanity checks for saliency maps with image and text classification
• A generalizable saliency map-based interpretation of model outcome
• Sanity Checks for Saliency Maps
• When saliency goes off on a tangent: Interpreting Deep Neural Networks with nonlinear saliency maps
• Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation