Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation\n Methods

2019/11/06 by Dylan Slack, Slack, Dylan, Sophie Hilgard +7 · 44 citations
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1911.02508

openalex publication_date 2019/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As machine learning black boxes are increasingly being deployed in domains\nsuch as healthcare and criminal justice, there is growing emphasis on building\ntools and techniques for explaining these black boxes in an interpretable\nmanner. Such explanations are being leveraged by domain experts to diagnose\nsystematic errors and underlying biases of black boxes. In this paper, we\ndemonstrate that post hoc explanations techniques that rely on input\nperturbations, such as LIME and SHAP, are not reliable. Specifically, we\npropose a novel scaffolding technique that effectively hides the biases of any\ngiven classifier by allowing an adversarial entity to craft an arbitrary\ndesired explanation. Our approach can be used to scaffold any biased classifier\nin such a way that its predictions on the input data distribution still remain\nbiased, but the post hoc explanations of the scaffolded classifier look\ninnocuous. Using extensive evaluation with multiple real-world datasets\n(including COMPAS), we demonstrate how extremely biased (racist) classifiers\ncrafted by our framework can easily fool popular explanation techniques such as\nLIME and SHAP into generating innocuous explanations which do not reflect the\nunderlying biases.\n

Citations

• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)

Cited by

• Few-Shot Learning of a Graph-Based Neural Network Model Without Backpropagation
• Where is the Watermark? Interpretable Watermark Detection at the Block Level
• Beyond Additivity: Sparse Isotonic Shapley Regression toward Nonlinear Explainability
• ABLE: Using Adversarial Pairs to Construct Local Models for Explaining Model Predictions
• Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian
• Interpretable Fine-Gray Deep Survival Model for Competing Risks: Predicting Post-Discharge Foot Complications for Diabetic Patients in Ontario
• Additive Large Language Models for Semi-Structured Text
• Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration
• Verified SHAP: Provable Bounds for Exact Shapley Values of Neural Networks
• Fair and Explainable Credit-Scoring under Concept Drift: Adaptive Explanation Frameworks for Evolving Populations
• Before the Clinic: Transparent and Operable Design Principles for Healthcare AI
• Feature-Function Curvature Analysis: A Geometric Framework for Explaining Differentiable Models
• Rule-Based Explanations for Retrieval-Augmented LLM Systems
• Tractable Shapley Values and Interactions via Tensor Networks
• SHAP Meets Tensor Networks: Provably Tractable Explanations with Parallelism
• Robust or Suggestible? Exploring Non-Clinical Induction in LLM Drug-Safety Decisions
• Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning
• Beyond single-model XAI: aggregating multi-model explanations for enhanced trustworthiness
• Denoised IPW-Lasso for Heterogeneous Treatment Effect Estimation in Randomized Experiments
• Explainable but Vulnerable: Adversarial Attacks on XAI Explanation in Cybersecurity Applications
• Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI
• Attack logics, not outputs: Towards efficient robustification of deep neural networks by falsifying concept-based properties
• Explainable machine learning for multiscale thermal conductivity modeling in polymer nanocomposites with uncertainty quantification
• Bridging human and machine intelligence: Reverse-engineering radiologist intentions for clinical trust and adoption
• Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?
• Smaller is Better: Enhancing Transparency in Vehicle AI Systems via Pruning
• Unifying local and global model explanations by functional decomposition of low dimensional structures
• Challenging common interpretability assumptions in feature attribution explanations
• Secure human oversight of AI: Threat modeling in a socio-technical context
• Interpretability as Alignment: Making Internal Understanding a Design Principle
• Evaluating the stability of model explanations in instance-dependent cost-sensitive credit scoring
• Extending LIME for Business Process Automation
• Delta-Audit: Explaining What Changes When Models Change
• Informative Post-Hoc Explanations Only Exist for Simple Functions
• X-Node: Self-Explanation is All We Need
• Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition
• Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks
• OrdShap: Feature Position Importance for Sequential Black-Box Models
• On the Effectiveness of Methods and Metrics for Explainable AI in Remote Sensing Image Scene Classification
• Securing AI Systems: A Guide to Known Attacks and Impacts
• VARSHAP: Addressing Global Dependency Problems in Explainable AI with Variance-Based Local Feature Attribution
• Model extraction from counterfactual explanations
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis

Related

• SHLIME: Foiling adversarial attacks fooling SHAP and LIME
• Improving LIME Robustness with Smarter Locality Sampling
• Feature Attributions and Counterfactual Explanations Can Be Manipulated
• Better sampling in explanation methods can prevent dieselgate-like deception
• Towards the Unification and Robustness of Perturbation and Gradient Based Explanations