LLMs Encode How Difficult Problems Are
2025/10/20 by William Lugoloobi, Chris Russell, Lugoloobi, William +1 · 11 voices
#cs.CL
paper · pdf · doi:10.48550/arxiv.2510.18147
Abstract
Large language models exhibit a puzzling inconsistency: they solve complex problems yet frequently fail on seemingly simpler ones. We investigate whether LLMs internally encode problem difficulty in a way that aligns with human judgment, and whether this representation tracks generalization during reinforcement learning post-training. We train linear probes across layers and token positions on 60 models, evaluating on mathematical and coding subsets of Easy2HardBench. We find that human-labeled difficulty is strongly linearly decodable (AMC: ρ≈ 0.88) and exhibits clear model-size scaling, whereas LLM-derived difficulty is substantially weaker and scales poorly. Steering along the difficulty direction reveals that pushing models toward "easier" representations reduces hallucination and improves accuracy. During GRPO training on Qwen2.5-Math-1.5B, the human-difficulty probe strengthens and positively correlates with test accuracy across training steps, while the LLM-difficulty probe degrades and negatively correlates with performance. These results suggest that human annotations provide a stable difficulty signal that RL amplifies, while automated difficulty estimates derived from model performance become misaligned precisely as models improve. We release probe code and evaluation scripts to facilitate replication.
Discussions
- LLMs encode how difficult problems are [hn, 174 points, 38 comments]
- https://arxiv.org/abs/2510.18147 LLMは難しい課題を解ける一方で簡単な問題で失敗することがあるため、研究者はLLM内部の「問題の難易度」について調査。 「人間が付けたラベル」と「モデル自身が示す難易度」を比較した結果、人間ラベルの難易度は高精度に読み取れるが、モデル自身の難易度は不安定だった。 強化学習後の解析では、人間の難易度表現は性能向上と正の相関を示した一方 [bsky, 1 points, 0 comments]
- https://bsky.app/profile/hackernews.com.web.brid.gy/post/3m4yx7ht6pka2 [bsky, 0 points, 0 comments]
- LLMs Encode How Difficult Problems Are [bsky, 0 points, 0 comments]
- LLMs Encode How Difficult Problems Are #HackerNews https://arxiv.org/abs/2510.18147 [bsky, 0 points, 0 comments]
- LLMs Encode How Difficult Problems Are https://arxiv.org/abs/2510.18147 (https://news.ycombinator.com/item?id=45838564) [bsky, 0 points, 0 comments]
- LLMs Encode How Difficult Problems Are https://arxiv.org/abs/2510.18147 [bsky, 0 points, 0 comments]
- LLMs encode how difficult problems are https://arxiv.org/abs/2510.18147 [comments] [132 points] [bsky, 0 points, 0 comments]
- That’s a compelling observation. Understanding *how* LLMs represent problem complexity could unlock significant gains in AI efficiency and reasoning. 🤖 #ai LLMs encode how difficult problems are [bsky, 0 points, 0 comments]
- LLMs encode how difficult problems are https://arxiv.org/abs/2510.18147 https://news.ycombinator.com/item?id=45838564 [bsky, 0 points, 0 comments]
- LLMs Encode How Difficult Problems Are https://arxiv.org/abs/2510.18147 (https://news.ycombinator.com/item?id=45838564) [bsky, 0 points, 0 comments]
Related