Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning

2024/10/06 by Shramay Palta, Palta, Shramay, Nishant Balepur +9 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Logic, Reasoning, and Knowledge

paper · pdf · doi:10.48550/arxiv.2410.10854

openalex publication_date 2024/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Questions involving commonsense reasoning about everyday situations often admit many possible or plausible answers. In contrast, multiple-choice question (MCQ) benchmarks for commonsense reasoning require a hard selection of a single correct answer, which, in principle, should represent the most plausible answer choice. On 250 MCQ items sampled from two commonsense reasoning benchmarks, we collect 5,000 independent plausibility judgments on answer choices. We find that for over 20% of the sampled MCQs, the answer choice rated most plausible does not match the benchmark gold answers; upon manual inspection, we confirm that this subset exhibits higher rates of problems like ambiguity or semantic mismatch between question and answer choices. Experiments with LLMs reveal low accuracy and high variation in performance on the subset, suggesting our plausibility criterion may be helpful in identifying more reliable benchmark items for commonsense evaluation.

Cited by

• Social Story Frames: Contextual Reasoning about Narrative Intent and Reception
• The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts
• Everything is Plausible: Investigating the Impact of LLM Rationales on Human Notions of Plausibility
• A large-scale evaluation of commonsense knowledge in humans and large language models
• Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It

Related

• LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning
• CASE: Commonsense-Augmented Score with an Expanded Answer Space
• Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
• ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
• Do Fine-tuned Commonsense Language Models Really Generalize?