From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

2025/06/14 by Bin Xie, Bingbing Xu, Xie, Bin +7 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Healthcare #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.12446

openalex publication_date 2025/06/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Inference-time alignment methods have gained significant attention for their efficiency and effectiveness in aligning large language models (LLMs) with human preferences. However, existing dominant approaches using reward-guided search (RGS) primarily rely on outcome reward models (ORMs), which suffer from a critical granularity mismatch: ORMs are designed to provide outcome rewards for complete responses, while RGS methods rely on process rewards to guide the policy, leading to inconsistent scoring and suboptimal alignment. To address this challenge, we introduce process reward models (PRMs) into RGS and argue that an ideal PRM should satisfy two objectives: Score Consistency, ensuring coherent evaluation across partial and complete responses, and Preference Consistency, aligning partial sequence assessments with human preferences. Based on these, we propose SP-PRM, a novel dual-consistency framework integrating score consistency-based and preference consistency-based partial evaluation modules without relying on human annotation. Extensive experiments on dialogue, summarization, and reasoning tasks demonstrate that SP-PRM substantially enhances existing RGS methods, achieving a 3.6%-10.3% improvement in GPT-4 evaluation scores across all tasks.

Citations

• Inference-time Alignment in Continuous Space
• SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
• Fast Best-of-N Decoding via Speculative Rejection
• GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment
• Inference-Time Language Model Alignment via Integrated Value Guidance
• Cascade Reward Sampling for Efficient Decoding-Time Alignment
• Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
• Value Augmented Sampling for Language Model Alignment and Personalization
• ARGS: Alignment as Reward-Guided Search
• Secrets of RLHF in Large Language Models Part I: PPO
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• LLaMA: Open and Efficient Foundation Language Models
• Chain of Hindsight Aligns Language Models with Feedback
• Training language models to follow instructions with human feedback
• A Contrastive Framework for Neural Text Generation
• Training Verifiers to Solve Math Word Problems
• Learning to summarize from human feedback
• Fine-Tuning Language Models from Human Preferences
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• Aligning Large Language Models with Human: A Survey

Cited by

• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
• GradeSQL: Test-Time Inference with Outcome Reward Models for Text-to-SQL Generation from Large Language Models
• AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Models

Related

• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
• R-PRM: Reasoning-Driven Process Reward Modeling
• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
• Rethinking Reward Models for Multi-Domain Test-Time Scaling
• SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins