Adversarial NLI: A New Benchmark for Natural Language Understanding

2019/10/31 by Yixin Nie, Adina Williams, Nie, Yixin +9 · 1 voice · 57 citations
Computer Science · #Adversarial Robustness in Machine Learning #Multimodal Machine Learning Applications #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1910.14599

openalex publication_date 2019/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

Citations

• Gradient-based learning applied to document recognition
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Multi-Task Deep Neural Networks for Natural Language Understanding
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• VQA: Visual Question Answering
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• A large annotated corpus for learning natural language inference
• Learning the Difference that Makes a Difference with Counterfactually-Augmented Data

Cited by

• Beyond Context: Large Language Models Failure to Grasp Users Intent
• FaithLens: Detecting and Explaining Faithfulness Hallucination
• MAGIC: Achieving Superior Model Merging via Magnitude Calibration
• Do Generalisation Results Generalise?
• Trification: A Comprehensive Tree-based Strategy Planner and Structural Verification for Fact-Checking
• A Rosetta Stone for AI Benchmarks
• Don't Learn, Ground: A Case for Natural Language Inference with Visual Grounding
• Fantastic Bugs and Where to Find Them in AI Benchmarks
• SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification
• PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise
• Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering
• LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
• Analyzing and Mitigating Negation Artifacts using Data Augmentation for Improving ELECTRA-Small Model Accuracy
• A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
• Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens
• Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text
• Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations
• MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference
• The Gray Zone of Faithfulness: Taming Ambiguity in Unfaithfulness Detection
• Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings
• Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs
• A Survey on Evaluation of Large Language Models
• Data-Model Co-Evolution: Growing Test Sets to Refine LLM Behavior
• F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data
• SocialNLI: A Dialogue-Centric Social Inference Dataset
• Large Language Models Hallucination: A Comprehensive Survey
• RL-Guided Data Selection for Language Model Finetuning
• Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Data
• The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
• Extractive Fact Decomposition for Interpretable Natural Language Inference in one Forward Pass
• On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
• DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction
• A Dynamic Knowledge Update-Driven Model with Large Language Models for Fake News Detection
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
• Learning from Diverse Reasoning Paths with Routing and Collaboration
• CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor
• Language Models are Few-Shot Learners
• EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint
• Recipes for Safety in Open-domain Chatbots
• Unlocking the Effectiveness of LoRA-FP for Seamless Transfer Implantation of Fingerprints in Downstream Models
• GIER: Gap-Driven Self-Refinement for Large Language Models
• Adversarial Training for Large Neural Language Models
• DELIVER: A System for LLM-Guided Coordinated Multi-Robot Pickup and Delivery using Voronoi-Based Relay Planning
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering
• Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
• Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
• Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators
• TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs
• FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing
• Semantic Encryption: Secure and Effective Interaction with Cloud-based Large Language Models via Semantic Transformation
• VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI
• FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal

Discussions

• Adversarial NLI: A New Benchmark for Natural Language Understanding [hn, 11 points, 0 comments]

Related

• ANLIzing the Adversarial Natural Language Inference Dataset
• Adversarial Analysis of Natural Language Inference Systems
• Teaching a Language Model to Distinguish Between Similar Details using a Small Adversarial Training Set
• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models