ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models

2025/02/03 by Zixuan Wu, Wu, Zixuan, Francesca Lucchetti +13 · 12 voices · 4 citations
Computer Science · Psychology · #Cognitive science #Computer science #Natural Language Processing Techniques #Natural language processing #Psychology #Semantic Web and Ontologies #Topic Modeling #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.01584

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/02/03 · openalex created_date 2025/02/06 · openalex updated_date 2026/08/04

Abstract

Existing benchmarks for frontier models often test specialized, "PhD-level" knowledge that is difficult for non-experts to grasp. In contrast, we present a benchmark with 613 problems based on the NPR Sunday Puzzle Challenge that requires only general knowledge. Our benchmark is challenging for both humans and models; however correct solutions are easy to verify, and models' mistakes are easy to spot. As LLMs are more widely deployed in society, we believe it is useful to develop benchmarks for frontier models that humans can understand without the need for deep domain expertise. Our work reveals capability gaps that are not evident in existing benchmarks: OpenAI o1 significantly outperforms other reasoning models on our benchmark, despite being on par with other models when tested on benchmarks that test specialized knowledge. Furthermore, our analysis of reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance, often concedes with "I give up" before providing an answer that it knows is wrong. R1 can also be remarkably "uncertain" in its output and in rare cases, it does not "finish thinking," which suggests the need for techniques to ``wrap up'' before the context window limit is reached. We also quantify the effectiveness of reasoning longer to identify the point beyond which more reasoning is unlikely to improve accuracy on our benchmark.

Citations

• LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing
• s1: Simple test-time scaling
• Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs

Cited by

• Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs
• NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problems
• Lexical Hints of Accuracy in LLM Reasoning Chains
• Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
• When Reasoning Beats Scale: A 1.5B Reasoning Model Outranks 13B LLMs as Discriminator

Discussions

• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models [hn, 174 points, 80 comments]
• 🧠 𝗣𝗵𝗗 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗡𝗼𝘁 𝗥𝗲𝗾𝘂𝗶𝗿𝗲𝗱: 𝗔 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝗳𝗼𝗿 𝗟𝗟𝗠𝘀 🤖 A new benchmark reveals gaps in reasoning skills of LLMs, with OpenAI leading at 59% a [bsky, 2 points, 0 comments]
• "PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models" Many people feel calling this a reasoning test is wrong. It's more about memory than smart thinking. AI is failing to see [bsky, 0 points, 0 comments]
• https://bsky.app/profile/hackernews.com.web.brid.gy/post/3lhrsegppbsm2 [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models https://arxiv.org/abs/2502.01584 (https://news.ycombinator.com/item?id=42992336) [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models https://arxiv.org/abs/2502.01584 [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models https://arxiv.org/abs/2502.01584 [comments] [16 points] [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models https://arxiv.org/abs/2502.01584 https://news.ycombinator.com/item?id=42992336 [bsky, 0 points, 0 comments]
• PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models https://arxiv.org/abs/2502.01584 (https://news.ycombinator.com/item?id=42992336) [bsky, 0 points, 0 comments]

Related

• LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation
• Large Language Models' Reasoning Stalls: An Investigation into the Capabilities of Frontier Models
• The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
• Large Language Models Imitate Logical Reasoning, but at what Cost?
• From System 1 to System 2: A Survey of Reasoning Large Language Models