Understanding Neural Networks through Representation Erasure

2016/12/24 by Jiwei Li, Li, Jiwei, Will Monroe +3 · 33 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1612.08220

openalex publication_date 2016/12/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

Citations

• Intriguing properties of neural networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence Level Training with Recurrent Neural Networks
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Sparse Overcomplete Word Vector Representations
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Cited by

• Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols
• Investigating Training and Generalization in Faithful Self-Explanations of Large Language Models
• Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian
• How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer
• Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation
• Beyond Over-Refusal: Scenario-Based Diagnostics and Post-Hoc Mitigation for Exaggerated Refusals in LLMs
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?
• From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
• IndiSeek learns information-guided disentangled representations
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• Class-Aware Reinforcement Learning for Counterfactual Explanation Generation
• Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations
• Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action Recognition
• Neural Image Compression and Explanation
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
• Trigger Where It Hurts: Unveiling Hidden Backdoors through Sensitivity with Sensitron
• Interpretable Question Answering on Knowledge Bases and Text
• Towards Interrogating Discriminative Machine Learning Models
• Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
• Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding
• Identify Susceptible Locations in Medical Records via Adversarial Attacks on Deep Predictive Models
• Human-grounded Evaluations of Explanation Methods for Text\n Classification
• What made you do this? Understanding black-box decisions with sufficient\n input subsets
• Investigating how well contextual features are captured by bi-directional recurrent neural network models
• Making Document-Level Information Extraction Right for the Right Reasons
• Second-Order Word Embeddings from Nearest Neighbor Topological Features
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
• Understanding Memory Modules on Learning Simple Algorithms
• What Context Features Can Transformer Language Models Use?

Related

• Interpretation of NLP models through input marginalization
• ERASER: A Benchmark to Evaluate Rationalized NLP Models
• Analyzing machine-learned representations: A natural language case study
• Analyzing and Interpreting Convolutional Neural Networks in NLP
• Evaluating Recurrent Neural Network Explanations