RM-R1: Reward Modeling as Reasoning

2025/05/05 by Xiusi Chen, Chen, Xiusi, Guojian Li +21 · 76 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Recommender Systems and Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.02387

openalex publication_date 2025/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reward modeling is essential for aligning large language models with human preferences through reinforcement learning. To provide accurate reward signals, a reward model (RM) should stimulate deep thinking and conduct interpretable reasoning before assigning a score or a judgment. Inspired by recent advances of long chain-of-thought on reasoning-intensive tasks, we hypothesize and validate that integrating reasoning into reward modeling significantly enhances RM's interpretability and performance. We introduce a new class of generative reward models, Reasoning Reward Models (ReasRMs), which formulate reward modeling as a reasoning task. We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1. RM-R1 features a chain-of-rubrics (CoR) mechanism -- self-generating sample-level chat rubrics or math/code solutions, and evaluating candidate responses against them. The training of RM-R1 consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learning with verifiable rewards. Empirically, our models achieve superior performance across three reward model benchmarks on average, outperforming much larger open-weight models (e.g., INF-ORM-Llama3.1-70B) and proprietary ones (e.g., GPT-4o) by up to 4.9%. Beyond final performance, we perform thorough analyses to understand the key ingredients of successful ReasRM training.

Citations

• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future
• Inference-Time Scaling for Generalist Reward Modeling
• JudgeLRM: Large Reasoning Models as a Judge
• ToRL: Scaling Tool-Integrated RL
• Process Reinforcement through Implicit Rewards
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Qwen2.5 Technical Report
• GPT-4o System Card
• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Self-Taught Evaluators
• The Llama 3 Herd of Models
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• Nemotron-4 340B Technical Report
• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
• Advancing LLM Reasoning Generalists with Preference Trees
• InternLM2 Technical Report
• RewardBench: Evaluating Reward Models for Language Modeling
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Bridging Language and Items for Retrieval and Recommendation
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• DeLLMa: Decision Making Under Uncertainty with Large Language Models
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
• HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Let's Verify Step by Step
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GPT-4 Technical Report
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Does Knowledge Distillation Really Work?
• Learning to summarize from human feedback
• Scaling Laws for Neural Language Models
• Fine-Tuning Language Models from Human Preferences
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
• The Expressive Power of Transformers with Chain of Thought

Cited by

• VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE
• When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models
• Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques
• From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• Think Twice: Branch-and-Rethink Reasoning Reward Model
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• Scalable Supervising Software Agents with Patch Reasoner
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling
• Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures
• Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models
• Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systems
• From to : Multidimensional Supervision of Reasoning Process for LLM Optimization
• A Survey on Agentic Multimodal Large Language Models
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning
• mR3: Multilingual Rubric-Agnostic Reward Reasoning Models
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
• Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensembling
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
• Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
• Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoning
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• Self-Aligned Reward: Towards Effective and Efficient Reasoners
• GRAM-R²: Self-Training Generative Foundation Reward Models for Reward Reasoning
• Reasoning-Intensive Regression
• Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
• Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs
• StepWiser: Stepwise Generative Judges for Wiser Reasoning
• PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning
• Expert Preference-based Evaluation of Automated Related Work Generation
• StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models
• VRPRM: Process Reward Modeling via Visual Reasoning
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• Libra: Assessing and Improving Reward Model by Learning to Think
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
• OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
• Pre-Trained Policy Discriminators are General Reward Models
• RLPR: Extrapolating RLVR to General Domains without Verifiers
• ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models
• GRAM: A Generative Foundation Reward Model for Reward Generalization
• Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks
• Scaling Test-time Compute for LLM Agents
• Schema-R1: A reasoning training approach for schema linking in Text-to-SQL Task
• Saffron-1: Safety Inference Scaling
• RewardAnything: Generalizable Principle-Following Reward Models
• Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
• Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning
• ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mind
• SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
• Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
• Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Models
• Think-J: Learning to Think for Generative LLM-as-a-Judge
• Reward Reasoning Model
• R3: Robust Rubric-Agnostic Reward Models
• Time-R1: Towards Comprehensive Temporal Reasoning in LLMs
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
• WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks
• Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
• Reward Modeling for Scientific Writing Evaluation
• Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
• ToolRL: Reward is All Tool Learning Needs

Related

• Reward Reasoning Model
• Libra: Assessing and Improving Reward Model by Learning to Think
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning