Reasoning Models Reason Well, Until They Don't
2025/10/25 by Revanth Rameshkumar, Rameshkumar, Revanth, Jimson Huang +7 · 14 voices
#cs.AI #cs.CL
paper · pdf · doi:10.48550/arxiv.2510.22371
Abstract
Large language models (LLMs) have shown significant progress in reasoning tasks. However, recent studies show that transformers and LLMs fail catastrophically once reasoning problems exceed modest complexity. We revisit these findings through the lens of large reasoning models (LRMs) -- LLMs fine-tuned with incentives for step-by-step argumentation and self-verification. LRM performance on graph and reasoning benchmarks such as NLGraph seem extraordinary, with some even claiming they are capable of generalized reasoning and innovation in reasoning-intensive fields such as mathematics, physics, medicine, and law. However, by more carefully scaling the complexity of reasoning problems, we show existing benchmarks actually have limited complexity. We develop a new dataset, the Deep Reasoning Dataset (DeepRD), along with a generative process for producing unlimited examples of scalable complexity. We use this dataset to evaluate model performance on graph connectivity and natural language proof planning. We find that the performance of LRMs drop abruptly at sufficient complexity and do not generalize. We also relate our LRM results to the distributions of the complexities of large, real-world knowledge graphs, interaction graphs, and proof datasets. We find the majority of real-world examples fall inside the LRMs' success regime, yet the long tails expose substantial failure potential. Our analysis highlights the near-term utility of LRMs while underscoring the need for new methods that generalize beyond the complexity of examples in the training distribution.
Citations
Discussions
- Reasoning models reason well, until they don't [hn, 218 points, 217 comments]
- Recent studies show that transformers and LLMs fail catastrophically once reasoning problems exceed modest complexity: arxiv.org/abs/2510.22371 [bsky, 2 points, 0 comments]
- LLM Reasoning Models Reason Well, Until They Don't arxiv.org/abs/2510.22371 * LLM: sig. progress in reasoning tasks * fail catastrophically once reasoning problems exceed modest complexity Reasoning m [bsky, 1 points, 1 comments]
- Reasoning Models Reason Well, Until They Don't https://arxiv.org/abs/2510.22371 https://news.ycombinator.com/item?id=45769971 [bsky, 0 points, 0 comments]
- Reasoning Models Reason Well, Until They Don't https://arxiv.org/abs/2510.22371 [bsky, 0 points, 0 comments]
- 📰 Reasoning Models Reason Well, Until They Don't 🔗 https://arxiv.org/abs/2510.22371 💬 Discuss on HN [bsky, 0 points, 0 comments]
- Reasoning Models Reason Well, Until They Don't https://arxiv.org/abs/2510.22371 [bsky, 0 points, 0 comments]
- Reasoning Models Reason Well, Until They Don't https://arxiv.org/abs/2510.22371 (https://news.ycombinator.com/item?id=45769971) [bsky, 0 points, 0 comments]
- https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3m4igcomgrqz2 [bsky, 0 points, 0 comments]
- Reasoning models reason well, until they don't https://arxiv.org/abs/2510.22371 (http://news.ycombinator.com/item?id=45769971) [bsky, 0 points, 0 comments]
- Reasoning Models Reason Well, Until They Don't view on hacker news [bsky, 0 points, 0 comments]
- ⚡ Hackernews Top story: Reasoning Models Reason Well, Until They Don't [bsky, 0 points, 0 comments]
- Reasoning Models Reason Well, Until They Don't https://arxiv.org/abs/2510.22371 (https://news.ycombinator.com/item?id=45769971) [bsky, 0 points, 0 comments]
- Reasoning Models Reason Well, Until They Don't #HackerNews https://arxiv.org/abs/2510.22371 [bsky, 0 points, 0 comments]
Related