Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models

2025/09/30 by Runze Liu, Liu, Runze, Jiakang Wang +25 · 3 citations
Business, Management and Accounting · Computer Science · #Business Process Modeling and Analysis #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2509.26628

openalex publication_date 2025/09/30 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency.

Citations

• A Survey of Reinforcement Learning for Large Reasoning Models
• Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training
• ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• Agentic Reinforced Policy Optimization
• Group Sequence Policy Optimization
• Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
• First Return, Entropy-Eliciting Explore
• Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning
• Thought Anchors: Which LLM Reasoning Steps Matter?
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
• TreeRPO: Tree Relative Policy Optimization
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
• Skywork Open Reasoner 1 Technical Report
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
• Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• Understanding R1-Zero-Like Training: A Critical Perspective
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
• Process Reinforcement through Implicit Rewards
• Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
• Qwen2.5 Technical Report
• Free Process Rewards without Process Labels
• GPT-4o System Card
• Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
• OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
• VinePPO: Refining Credit Assignment in RL Training of LLMs
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GPT-4 Technical Report
• Solving math word problems with process- and outcome-based feedback
• Solving Quantitative Reasoning Problems with Language Models
• Attention Is All You Need
• Adam: A Method for Stochastic Optimization
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
• Reinforcement Learning: An Introduction

Cited by

• Robust Preference Alignment via Directional Neighborhood Consensus
• GraphTracer: Graph-Guided Failure Tracing in LLM Agents for Robust Multi-Turn Deep Search
• When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL

Related

• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
• Effective Reinforcement Learning for Reasoning in Language Models