Large Language Model Reasoning Failures

2026/02/05 by Peiyang Song, Pengrui Han, Noah Goodman · 37 voices · 2 citations
#cs.AI #cs.CL #cs.LG

paper · pdf

Abstract

Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios. To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in LLMs. We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into informal (intuitive) and formal (logical) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to LLM architectures that broadly affect downstream tasks; application-specific limitations that manifest in particular domains; and robustness issues characterized by inconsistent performance across minor variations. For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes, and present mitigation strategies. By unifying fragmented research efforts, our survey provides a structured perspective on systemic weaknesses in LLM reasoning, offering valuable insights and guiding future research towards building stronger, more reliable, and robust reasoning capabilities. We additionally release a comprehensive collection of research works on LLM reasoning failures, as a GitHub repository at https://github.com/Peiyang-Song/Awesome-LLM-Reasoning-Failures, to provide an easy entry point to this area.

Cited by

• Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
• Implicit Reasoning Steering via Concept Chaining
• QuiLL: An LLM-Based Vulnerability Assessment Framework for the Wild

Discussions

• Large Language Model Reasoning Failures [hn, 40 points, 82 comments]
• Great reference from @garymarcus.bsky.social to this comprehensive paper. arxiv.org/pdf/2602.06176 "Large Language Model Reasoning Failures" Besides many other important finding, this was pretty clear [bsky, 17 points, 5 comments]
• Die Studie zeigt, dass LLMs systematische Schwächen beim logischen, symbolischen und commonsense-basierten Schlussfolgern haben. Die Fehler entstehen vor allem durch Trainings- und Repräsentationsgren [bsky, 12 points, 0 comments]
• "Overall, the systematic study of reasoning failures in LLMs parallels fault-tolerance research in early computing and incident analysis in safety-critical industries: understanding and categorizing f [bsky, 8 points, 1 comments]
• Large Language Model Reasoning Failures [bsky, 7 points, 0 comments]
• arxiv.org/abs/2602.06176 [bsky, 6 points, 0 comments]
• You might find this recent "..comprehensive survey dedicated to reasoning failures in LLMs" interesting: arxiv.org/abs/2602.06176 [bsky, 2 points, 0 comments]
• things are gonna get ugly www.arxiv.org/abs/2602.06176 "Having been trained on extensive human-generated data, LLMs inevitably inherit embedded social and ethical biases from those data resourceS" PRE [bsky, 2 points, 1 comments]
• sharing a new preprint bc it seems like a lot of people could use it -- a literature review of LLM reasoning failures. i am covering this paper but that piece has not come out yet. the researchers' go [bsky, 2 points, 1 comments]
• Ich glaube, wenn so etwas berichtet würde, weil momentan alle am feiern sind, dann wäre das als eine Art Totalschaden zu sehen. [bsky, 2 points, 0 comments]
• Large Language Model Reasoning Failures https://arxiv.org/abs/2602.06176 [bsky, 1 points, 0 comments]
• They still can't reason reliably and never will. The problem isn't just is the useful stuff they do worth the harm, its the useful stuff can't possible justify the cap ex and there is therefore no gua [bsky, 1 points, 0 comments]
• LLM reasoning still has so many flaws that Stanford and Caltech researchers wrote an entire journal article categorizing them! 🤯 The paper is in Transactions on Machine learning Research, available o [bsky, 1 points, 1 comments]
• Your thread reminds me of this chart from the paper "Large Language Model Reasoning Failures" (doi.org/10.48550/arX...). It doesn't offer solutions, but goes through a good survey and mapping of reas [bsky, 1 points, 0 comments]
• Large Language Model Reasoning Failures [hn, 1 points, 0 comments]
• LLM Reasoning Failures [hn, 1 points, 0 comments]
• Large Language Model Reasoning Failures [hn, 1 points, 0 comments]
• Large Language Model Reasoning Failures [bsky, 0 points, 0 comments]
• Is it time to sell my AI stocks? [bsky, 0 points, 0 comments]
• i'm not the only one that believes reasoning is currently insufficient arxiv.org/abs/2602.06176 [bsky, 0 points, 1 comments]
• ⚡ Hackernews Top story: Large Language Model Reasoning Failures [bsky, 0 points, 0 comments]
• Large Language Model Reasoning Failures https://arxiv.org/abs/2602.06176 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2602.06176 [bsky, 0 points, 0 comments]
• How do they think and do they think in ways comparable to how humans do. Are they able to decipher right from wrong even if adversarial information is added to their training data. Why did researchers [bsky, 0 points, 1 comments]
• Large Language Model Reasoning Failures https://arxiv.org/abs/2602.06176 (https://news.ycombinator.com/item?id=47098839) [bsky, 0 points, 0 comments]
• Here's the proof. [bsky, 0 points, 0 comments]
• 2/ But let's just burn the #earth down with $1 trillion in 2026 alone building #AI #GenAI that DOESN'T work, see - arxiv.org/abs/2602.06176 Or spend a couple of trillion $ a year on #war #weapons #dem [bsky, 0 points, 0 comments]
• arxiv.org/abs/2602.06176 [bsky, 0 points, 0 comments]
• There is a recent survey of reasoning failures patterns in LLMs. They do not analyze individual models, but it would be cool to see how the various models benchmark on their taxonomy of reasoning fail [bsky, 0 points, 0 comments]
• Large Language Model Reasoning Failures https:// arxiv.org/abs/2602.06176 # arxiv [mastodon, 0 points, 0 comments]
• Large Language Model Reasoning Failures “We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into inform [bsky, 0 points, 0 comments]
• Elsewhere in The 'Verse: A little light reading today. Large Language Model Reasoning Failures Peiyang Song, Pengrui Han, Noah Goodman arxiv.org/pdf/2602.06176 [bsky, 0 points, 0 comments]
• Large Language Model Reasoning Failures https://arxiv.org/abs/2602.06176 [bsky, 0 points, 0 comments]
• Large Language Model Reasoning Failures https://arxiv.org/abs/2602.06176 [comments] [12 points] [bsky, 0 points, 0 comments]
• LLM Reasoning Failures Worth a read #siliconvalley #techbros #openai #LLM #AI arxiv.org/pdf/2602.06176 [bsky, 0 points, 0 comments]
• 5/ These failures represent a great opportunity to research how to overcome LLM limitations and improve applications built on their reasoning capabilities. Link to the paper: arxiv.org/pdf/2602.06176 [bsky, 0 points, 0 comments]
• Large Language Model Reasoning Failures arxiv.org/abs/2602.061... [bsky, 0 points, 0 comments]

Related

• Reasoning LLMs are Wandering Solution Explorers
• Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
• Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
• A NotSo Simple Way to Beat Simple Bench
• Case Study: Testing Model Capabilities in Some Reasoning Tasks