Adversarial Examples for Evaluating Reading Comprehension Systems

2017/07/23 by Robin Jia, Percy Liang, Jia, Robin +1 · 1 voice · 264 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Adversarial system #Adversary #Artificial intelligence #Comprehension #Computation and Language (cs.CL) #Computer science #Computer security #FOS: Computer and information sciences #Language model #Linguistics #Machine Learning (cs.LG) #Mathematics #Natural Language Processing Techniques #Natural language processing #Programming language #Reading (process) #Reading comprehension #Scheme (mathematics) #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1707.07328

published in arXiv (Cornell University) (Cornell University) · EMNLP 2017

arxiv created 2017/07/23 · openalex publication_date 2017/07/23 · arxiv updated 2017/07/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of 75% F1 score to 36%; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to 7%. We hope our insights will motivate the development of new models that understand language more precisely.

Citations

• Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
• Structural Embedding of Syntactic Trees for Machine Comprehension

Cited by

• Beyond Context: Large Language Models' Failure to Grasp Users' Intent
• Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols
• Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation
• DAMASHA: Detecting AI in Mixed Adversarial Texts via Segmentation with Human-interpretable Attribution
• Analyzing and Mitigating Negation Artifacts using Data Augmentation for Improving ELECTRA-Small Model Accuracy
• Cache Mechanism for Agent RAG Systems
• FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models
• CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models
• Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models
• Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering
• Adversarial Robustness in One-Stage Learning-to-Defer
• Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
• ConDABench: Interactive Evaluation of Language Models for Data Analysis
• Evaluating the Robustness of a Production Malware Detection System to Transferable Adversarial Attacks
• Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset
• SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds
• MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages
• Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder
• Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering
• Can NLI Models Verify QA Systems' Predictions?
• Adversarial Attack on Graph Structured Data
• How Causal Abstraction Underpins Computational Explanation
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Special-Character Adversarial Attacks on Open-Source Language Model
• Semantics Preserving Adversarial Learning
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification
• HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal
• ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
• The Importance of Being Recurrent for Modeling Hierarchical Structure
• Exploring Neural Models for Parsing Natural Language into First-Order Logic
• Adversarial machine learning :
• Verification of Neural Network Control Policy Under Persistent Adversarial Perturbation
• Counterfactual Variable Control for Robust and Interpretable Question Answering
• Efficient and Robust Question Answering from Minimal Context over Documents
• Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models
• A Survey on Neural Machine Reading Comprehension
• Robust Multilingual Part-of-Speech Tagging via Adversarial Training
• Visual Referring Expression Recognition: What Do Systems Actually Learn?
• Mitigating Shortcut Learning with InterpoLated Learning
• PredCoin: Defense against Query-based Hard-label Attack
• Answer Generation through Unified Memories over Multiple Passages
• An Empirical Evaluation of Adversarial Robustness under Transfer Learning
• Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
• Imbalanced Sentiment Classification Enhanced with Discourse Marker
• Towards Fast Computation of Certified Robustness for ReLU Networks
• Adversarial Attack Classification and Robustness Testing for Large Language Models for Code
• TAI3: Testing Agent Integrity in Interpreting User Intent
• Golden Grain: Building a Secure and Decentralized Model Marketplace for MLaaS
• Commonsense knowledge adversarial dataset that challenges ELECTRA
• Adversarial Examples: Attacks and Defenses for Deep Learning
• Reinforced Mnemonic Reader for Machine Reading Comprehension
• Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification
• SeqPE: Transformer with Sequential Position Encoding
• Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices
• Adversarial Attacks for Multi-view Deep Models
• Normative Conflicts and Shallow AI Alignment
• SUCEA: Reasoning-Intensive Retrieval for Adversarial Fact-checking through Claim Decomposition and Editing
• Interactive Language Learning by Question Answering
• Built-in Vulnerabilities to Imperceptible Adversarial Perturbations
• Daedalus: Breaking Non-Maximum Suppression in Object Detection via Adversarial Examples
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Adversarially Robust Generalization Requires More Data
• Artificial Intelligence Strategies for National Security and Safety Standards
• Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models
• Evaluating Gender Bias in Natural Language Inference
• TRAPDOC: Deceiving LLM Users by Injecting Imperceptible Phantom Tokens into Documents
• Knowing More About Questions Can Help: Improving Calibration in Question Answering
• Geometry matters: Exploring language examples at the decision boundary
• Unshuffling Data for Improved Generalization
• Tunneling Neural Perception and Logic Reasoning through Abductive Learning
• Evaluating the Retrieval Robustness of Large Language Models
• Generating Semantically Valid Adversarial Questions for TableQA
• Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning
• Exploring the Robustness of NMT Systems to Nonsensical Inputs
• On VLMs for Diverse Tasks in Multimodal Meme Classification
• Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models
• Robust Machine Comprehension Models via Adversarial Training
• Mapping to Declarative Knowledge for Word Problem Solving
• Metamorphic Relation Based Adversarial Attacks on Differentiable Neural Computer
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
• Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
• Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks
• SPIRIT: Patching Speech Language Models against Jailbreak Attacks
• Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors
• A Study of the Tasks and Models in Machine Reading Comprehension
• Can Edge Probing Tasks Reveal Linguistic Knowledge in QA Models?
• QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
• The Future of Misinformation Detection: New Perspectives and Trends
• Knowledge Based Machine Reading Comprehension
• An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
• Generating Natural Adversarial Examples
• Conditional BERT Contextual Augmentation
• The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents
• Design and Interpretation of Universal Adversarial Patches in Face Detection
• FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation
• Reevaluating Adversarial Examples in Natural Language
• On the Robustness of GUI Grounding Models Against Image Attacks
• TextBugger: Generating Adversarial Text Against Real-world Applications
• Explicit Utilization of General Knowledge in Machine Reading Comprehension
• aiXamine: Simplified LLM Safety and Security
• DANCin SEQ2SEQ: Fooling Text Classifiers with Adversarial Text Example Generation
• Validity, Reliability, and Transparency in Artificial Intelligence Regulation
• QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models
• Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions
• Towards Improving Adversarial Training of NLP Models
• Trustworthy AI Must Account for Interactions

Discussions

• One of the techniques to test LLM apparent reasoning is introducing unrelated information. The recent GSM Symbolic Apple paper does some of this. The original I was looking for was "Adversarial exampl [bsky, 1 points, 0 comments]

Related

• Power in Numbers: Robust reading comprehension by finetuning with four adversarial sentences per example
• The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models
• Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models
• Robust Machine Comprehension Models via Adversarial Training
• Trick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering