Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning

2025/07/02 by Wu Fei, Fei, Wu, Shuxian Liang +13 · 1 citation
#cs.LG #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2507.01551

Abstract

Process Reinforcement Learning~(PRL) has demonstrated considerable potential in enhancing the reasoning capabilities of Large Language Models~(LLMs). However, introducing additional process reward models incurs substantial computational overhead, and there is no unified theoretical framework for process-level advantage estimation. To bridge this gap, we propose Self-Guided Process Reward Optimization~(SPRO), a novel framework that enables process-aware RL through two key innovations: (1) we show that process rewards can be derived intrinsically from the policy model itself, and (2) we redefine step-wise advantage by introducing well-defined Cumulative Process Rewards~(CPR) and Masked Step Advantage~(MSA), which facilitates rigorous step-wise action advantage estimation within shared-prompt sampling groups. Our experimental results show that SPRO outperforms vanilla GRPO with 3.4x higher training efficiency and a 12.9% test accuracy improvement. Furthermore, SPRO maintains a stable and elevated policy entropy throughout training while achieving a considerable reduction in response length, evidencing sufficient exploration and prevention of reward hacking. Notably, SPRO incurs no additional computational overhead compared to outcome-supervised RL methods such as GRPO, which benefit industrial implementation.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Attention Is All You Need

Cited by

• Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models

Related

• Process Reinforcement through Implicit Rewards
• Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks
• Entropy-Regularized Process Reward Model
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
• SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression