Interpretable Explanations of Black Boxes by Meaningful Perturbation

2017/04/11 by Ruth Fong, Ruth C. Fong, Andrea Vedaldi · 1 voice · 6 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Explainable Artificial Intelligence (XAI) #cs.AI #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.1109/iccv.2017.371

published as Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV) · Final camera-ready paper published at ICCV 2017 (Supplementary materials: http://openaccess.thecvf.com/content_ICCV_2017/supplemental/Fong_Interpretable_Explanations_of_ICCV_2017_supplemental.pdf)

arxiv published 2017/04/11 · openalex created_date 2017/04/28 · openalex publication_date 2017/10/01 · arxiv created 2021/12/03 · arxiv updated 2021/12/06 · openalex updated_date 2026/08/02

Abstract

As machine learning algorithms are increasingly applied to high impact yet high risk tasks, such as medical diagnosis or autonomous driving, it is critical that researchers can explain how such algorithms arrived at their predictions. In recent years, a number of image saliency methods have been developed to summarize where highly complex neural networks "look" in an image for evidence for their predictions. However, these techniques are limited by their heuristic nature and architectural constraints. In this paper, we make two main contributions: First, we propose a general framework for learning different kinds of explanations for any black box algorithm. Second, we specialise the framework to find the part of an image most responsible for a classifier decision. Unlike previous works, our method is model-agnostic and testable because it is grounded in explicit and interpretable image perturbations.

Citations

• ImageNet Large Scale Visual Recognition Challenge

Cited by

• Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
• GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAs
• From Heatmaps to Structural Explanations of Image Classifiers
• CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification
• Ensembles of Convolutional Neural Network models for pediatric pneumonia diagnosis
• Toward Learning Model-Agnostic Explanations for Deep Learning-Based Signal Modulation Classifiers
• PCBEAR: Pose Concept Bottleneck for Explainable Action Recognition
• GAUGE: Granularity-Adaptive Counterfactual Gating of Evidence for Incomplete Multimodal Classification
• A Self-Explainable Deep Architecture for Security Applications
• Interactivity x Explainability: Toward Understanding How Interactivity Can Improve Computer Vision Explanations

Discussions

• Using meaningful perturbations to improve interpretability of neural networks [hn, 2 points, 0 comments]

Related

• Reinforcement Explanation Learning
• Relevant Irrelevance: Generating Alterfactual Explanations for Image Classifiers
• Sanity Checks for Saliency Maps
• What's in the box? Explaining the black-box model through an evaluation of its interpretable features
• Explanation by Progressive Exaggeration