Gradient-based Analysis of NLP Models is Manipulable

2020/10/12 by Junlin Wang, Jens Tuyls, Wang, Junlin +5 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2010.05419

openalex publication_date 2020/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness. In this paper, however, we demonstrate that the gradients of a model are easily manipulable, and thus bring into question the reliability of gradient-based analyses. In particular, we merge the layers of a target model with a Facade that overwhelms the gradients without affecting the predictions. This Facade can be trained to have gradients that are misleading and irrelevant to the task, such as focusing only on the stop words in the input. On a variety of NLP tasks (text classification, NLI, and QA), we show that our method can manipulate numerous gradient-based analysis techniques: saliency maps, input reduction, and adversarial perturbations all identify unimportant or targeted tokens as being highly important. The code and a tutorial of this paper is available at http://ucinlp.github.io/facade.

Citations

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)
• Improving the Adversarial Robustness and Interpretability of Deep Neural\n Networks by Regularizing their Input Gradients
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case

Cited by

• Logic Traps in Evaluating Attribution Scores

Related

• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
• Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models
• Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation
• MASE: Interpretable NLP Models via Model-Agnostic Saliency Estimation