RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge

2021/01/02 by Bill Yuchen Lin, Ziyi Wu, Lin, Bill Yuchen +7 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2101.00376

openalex publication_date 2021/01/02 · openalex created_date 2021/07/19 · openalex updated_date 2026/07/28

Abstract

Question: I have five fingers but I am not alive. What am I? Answer: a glove. Answering such a riddle-style question is a challenging cognitive process, in that it requires complex commonsense reasoning abilities, an understanding of figurative language, and counterfactual reasoning skills, which are all important abilities for advanced natural language understanding (NLU). However, there are currently no dedicated datasets aiming to test these abilities. Herein, we present RiddleSense, a new multiple-choice question answering task, which comes with the first large dataset (5.7k examples) for answering riddle-style commonsense questions. We systematically evaluate a wide range of models over the challenge, and point out that there is a large gap between the best-supervised model and human performance -- suggesting intriguing future research in the direction of higher-order commonsense reasoning and linguistic creativity towards building advanced NLU systems.

Citations

• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Cited by

• Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles
• The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddles
• CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge
• Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
• PHANTOM RECALL: When Familiar Puzzles Fool Smart Models
• PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzles
• Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Models
• AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
• Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training

Related

• BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles
• The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans
• RiddleBench: A New Generative Reasoning Benchmark for LLMs
• RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation
• How Additional Knowledge can Improve Natural Language Commonsense Question Answering?