TreeRPO: Tree Relative Policy Optimization

2025/06/05 by Zhicheng Yang, Zhijiang Guo, Yang, Zhicheng +9 · 16 citations
Economics, Econometrics and Finance · Engineering · #Artificial Intelligence (cs.AI) #Climate Change Policy and Economics #Electric Power System Optimization #FOS: Computer and information sciences #Game Theory and Voting Systems #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.05183

openalex publication_date 2025/06/05 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have shown remarkable reasoning capabilities through Reinforcement Learning with Verifiable Rewards (RLVR) methods. However, a key limitation of existing approaches is that rewards defined at the full trajectory level provide insufficient guidance for optimizing the intermediate steps of a reasoning process. To address this, we introduce \name, a novel method that estimates the mathematical expectations of rewards at various reasoning steps using tree sampling. Unlike prior methods that rely on a separate step reward model, \name directly estimates these rewards through this sampling process. Building on the group-relative reward training mechanism of GRPO, \name innovatively computes rewards based on step-level groups generated during tree sampling. This advancement allows \name to produce fine-grained and dense reward signals, significantly enhancing the learning process and overall performance of LLMs. Experimental results demonstrate that our \name algorithm substantially improves the average Pass@1 accuracy of Qwen-2.5-Math on test benchmarks, increasing it from 19.0% to 35.5%. Furthermore, \name significantly outperforms GRPO by 2.9% in performance while simultaneously reducing the average response length by 18.1%, showcasing its effectiveness and efficiency. Our code will be available at \hrefhttps://github.com/yangzhch6/TreeRPOhttps://github.com/yangzhch6/TreeRPO.

Cited by

• TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models
• SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagery
• CARL: Criticality-Aware Agentic Reinforcement Learning
• Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewards
• Reasoning-Aware GRPO using Process Mining
• Scheduling Your LLM Reinforcement Learning with Reasoning Trees
• XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
• Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
• Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO
• Tree Search for LLM Agent Reinforcement Learning
• GRPO is Secretly a Process Reward Model
• MAPO: Mixed Advantage Policy Optimization
• Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
• Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning
• Towards a Unified View of Large Language Model Post-Training
• Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

Related

• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
• Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
• TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
• Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards