Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding

2025/09/08 by Li, Ziheng, Sun, Zexu, Zhao, Jinman +8 · 3 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.06923

Abstract

Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability. LLMs fail to discover viable reasoning paths when problems are overly difficult, while learning little new capability when problems are too simple. In this work, we formalize the impact of problem difficulty by quantifying the relationship between loss descent speed and rollout accuracy. Building on this analysis, we propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region. SEELE augments each training sample by appending a hint (part of a full solution) after the original problem. Unlike previous hint-based approaches, SEELE deliberately and adaptively adjusts the hint length for each problem to achieve an optimal difficulty. To determine the optimal hint length, SEELE employs a multi-round rollout sampling strategy. In each round, it fits an item response theory model to the accuracy-hint pairs collected in preceding rounds to predict the required hint length for the next round. This instance-level, real-time difficulty adjustment aligns problem difficulty with the evolving model capability, thereby improving exploration efficiency. Experimental results show that SEELE outperforms Group Relative Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +11.8 and +10.5 points, respectively, and surpasses the best previous supervision-aided approach by +3.6 points on average across six math reasoning benchmarks.

Citations

• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason
• Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
• SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
• Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration
• UFT: Unifying Supervised and Reinforcement Fine-Tuning
• TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example
• LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
• Learning to Reason under Off-Policy Guidance
• Improving RL Exploration for LLM Reasoning through Retrospective Replay
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
• Boosting MLLM Reasoning with Text-Debiased Hint-GRPO
• A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
• Understanding R1-Zero-Like Training: A Critical Perspective
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• Solving Quantitative Reasoning Problems with Language Models
• Training Verifiers to Solve Math Word Problems
• Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement
• Measuring Mathematical Problem Solving With the MATH Dataset
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Cited by

• ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
• Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning

Related

• Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
• StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• Hint-Guided Diversified Policy Optimization for LLM Reasoning
• GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning