Value-Guided Search for Efficient Chain-of-Thought Reasoning

2025/05/23 by Wang, Kaiwen, Zhou, Jin Peng, Chang, Jonathan +4 · 5 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.17373

Abstract

In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.

Cited by

• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
• Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts
• TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
• MAVIS: Multi-Objective Alignment via Value-Guided Inference-Time Search

Related

• Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence
• Language Models can Self-Improve at State-Value Estimation for Better Search
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling