Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding

2025/06/17 by So, Yeonkyoung, Lee, Gyuseong, Jung, Sungmok +4 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.14397

Abstract

Negation is a fundamental linguistic phenomenon that poses persistent challenges for Large Language Models (LLMs), particularly in tasks requiring deep semantic understanding. Existing benchmarks often treat negation as a side case within broader tasks like natural language inference, resulting in a lack of benchmarks that exclusively target negation understanding. In this work, we introduce Thunder-NUBench, a novel benchmark explicitly designed to assess sentence-level negation understanding in LLMs. Thunder-NUBench goes beyond surface-level cue detection by contrasting standard negation with structurally diverse alternatives such as local negation, contradiction, and paraphrase. The benchmark consists of manually curated sentence-negation pairs and a multiple-choice dataset that enables in-depth evaluation of models' negation understanding.

Citations

• Gemma 3 Technical Report
• Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
• Vision-Language Models Do Not Understand Negation
• Qwen2.5 Technical Report
• The Self-Contained Negation Test Set
• The Llama 3 Herd of Models
• How and where does CLIP process negation?
• Paraphrasing in Affirmative Terms Improves Negation Understanding
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• This is not correct! Negation-aware Evaluation of Language Generation Systems
• A negation detection assessment of GPTs: analysis with the xNot360 dataset
• Language models are not naysayers: An analysis of language models on negation benchmarks
• Better Zero-Shot Reasoning with Self-Adaptive Prompting
• Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
• Leveraging Affirmative Interpretations from Negation Improves Natural Language Understanding
• Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
• A Question-Answer Driven Approach to Reveal Affirmative Interpretations from Verbal Negations
• An Analysis of Negation in Natural Language Understanding Corpora
• Training Verifiers to Solve Math Word Problems
• A comparative evaluation and analysis of three generations of Distributional Semantic Models
• Understanding by Understanding Not: Modeling Negation in Language Models
• "I'm Not Mad": Commonsense Implications of Negation and Contradiction
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
• Language Models are Few-Shot Learners
• NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution
• Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
• What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Can neural networks understand monotonicity reasoning?
• HellaSwag: Can a Machine Really Finish Your Sentence?
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Stress Test Evaluation for Natural Language Inference
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• An Introduction to Convolutional Neural Networks
• A large annotated corpus for learning natural language inference
• An Introduction to Conditional Random Fields
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Cited by

• Don't Think of the White Bear: Ironic Negation in Transformer Models Under Cognitive Load

Related

• This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models
• Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation
• Negation: A Pink Elephant in the Large Language Models' Room?
• Making Language Models Robust Against Negation
• Language models are not naysayers: An analysis of language models on negation benchmarks