Explanations can be manipulated and geometry is to blame

2019/06/19 by Ann-Kathrin Dombrowski, Dombrowski, Ann-Kathrin, Maximilian Alber +11 · 34 citations
Computer Science · Decision Sciences · Mathematics · Medicine · #Artificial Intelligence in Healthcare and Education #Cryptography and Security (cs.CR) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Scientific Computing and Data Management #cs.CR #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1906.07983

openalex publication_date 2019/06/19 · arxiv created 2019/09/25 · arxiv updated 2019/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• ImageNet classification with deep convolutional neural networks
• Interpretation of Neural Networks is Fragile
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
• ϕ-Table: A Statistical Explanation for Global SHAP
• Rethinking Robustness: A New Approach to Evaluating Feature Attribution Methods
• MASE: Interpretable NLP Models via Model-Agnostic Saliency Estimation
• Formal Abductive Latent Explanations for Prototype-Based Networks
• Disentangling Neurodegeneration with Brain Age Gap Prediction Models: A Graph Signal Processing Perspective
• Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning
• Prime Implicant Explanations for Reaction Feasibility Prediction
• Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI
• "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
• Quantitative Evaluations on Saliency Methods: An Experimental Study
• Robust and Stable Black Box Explanations
• Trustworthy Convolutional Neural Networks: A Gradient Penalized-based Approach
• BMB-LIME: LIME with modeling local nonlinearity and uncertainty in explainability
• Transparency of medical artificial intelligence systems
• Best of both worlds: local and global explanations with human-understandable concepts
• Attribution Explanations for Deep Neural Networks: A Theoretical Perspective
• Pulling Back the Curtain on Deep Networks
• A Game-Theoretic Taxonomy of Visual Concepts in DNNs
• DiffGradCAM: A Class Activation Map Using the Full Model Decision to Solve Unaddressed Adversarial Attacks
• Securing AI Systems: A Guide to Known Attacks and Impacts
• Model Guidance via Robust Feature Attribution
• Pixel-level Certified Explanations via Randomized Smoothing
• Model extraction from counterfactual explanations
• Aggregating explanation methods for stable and robust explainability
• Robustness questions the interpretability of graph neural networks: what to do?
• Misaligning Reasoning with Answers -- A Framework for Assessing LLM CoT Robustness
• Fixed Point Explainability
• Measuring Explainer Stability via Attribution Separability
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• DiCE-Extended: A Robust Approach to Counterfactual Explanations in Machine Learning
• Learning Fair Rule Lists
• Pitfalls in machine learning interpretability: Manipulating partial dependence plots to hide discrimination
• A Self-Explainable Deep Architecture for Security Applications
• Uncovering the Structure of Explanation Quality with Spectral Analysis
• Evaluating the robustness of explainable AI in medical image recognition under natural and adversarial data corruption

Related

• Towards Robust Explanations for Deep Neural Networks
• Do Explanations Explain? Model Knows Best
• Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance
• Aggregating explanation methods for stable and robust explainability
• Explaining Neural Networks with Reasons