Inference-Time Scaling for Generalist Reward Modeling

2025/04/03 by Zijun Liu, Liu, Zijun, Wang Peiyi +16 · 18 voices · 72 citations
Computer Science · #Topic Modeling #Machine Learning in Healthcare #Explainable Artificial Intelligence (XAI)

paper · pdf · doi:10.48550/arxiv.2504.02495

Abstract

Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• The Lessons of Developing Process Reward Models in Mathematical Reasoning
• Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
• Self-Rewarding Language Models
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Learning to summarize from human feedback

Cited by

• Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
• AI Can Learn Scientific Taste
• Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family
• RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• More Than the Final Answer: Improving Visual Extraction and Logical Consistency in Vision-Language Models
• From Show Programmes to Data: Designing a Workflow to Make Performing Arts Ephemera Accessible Through Language Models
• PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• GTM: Simulating the World of Tools for AI Agents
• Highly Efficient Test-Time Scaling for T2I Diffusion Models with Text Embedding Perturbation
• Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
• CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
• Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques
• Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving
• From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
• DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF
• SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
• The Peril of Preference: Why GRPO fails on Ordinal Rewards
• The Sequential Edge: Inverse-Entropy Voting Beats Parallel Self-Consistency at Matched Compute
• One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning
• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• Think Twice: Branch-and-Rethink Reasoning Reward Model
• Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling
• Dual-Weighted Reinforcement Learning for Generative Preference Modeling
• Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systems
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• Online Rubrics Elicitation from Pairwise Comparisons
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
• SITCOM: Scaling Inference-Time COMpute for VLAs
• From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling
• Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks
• Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
• EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Think Socially via Cognitive Reasoning
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration
• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
• RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
• Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solving
• Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Models
• Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference
• RewardDance: Reward Scaling in Visual Generation
• Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
• In2x at WMT25 Translation Task
• Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
• Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
• PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning
• CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization
• From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
• Expert Preference-based Evaluation of Automated Related Work Generation
• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
• Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillation
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• RecGPT Technical Report
• Libra: Assessing and Improving Reward Model by Learning to Think
• DeepSeek (chatbot) [wikipedia]

Discussions

• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling [hn, 163 points, 35 comments]
• Well the latest DeepSeek is very satisfying from an humanities perspective. The trick to generalize RL is replacing scalar grades with… source criticism (qualitative principles and critiques). arxiv.o [bsky, 45 points, 0 comments]
• 🚨New DeepSeek Model Incoming🚨 but first they release the paper describing generative reward modeling (GRM) via Self-Principled Critique Tuning (SPCT) looking forward to DeepSeek-GRM! arxiv.org/abs/2 [bsky, 30 points, 4 comments]
• Looks like DeepSeek is also now on the Inference time scaling for reward models train arxiv.org/abs/2504.02495 with a cute diagram comparing it to our CLoud rewards arxiv.org/abs/2408.11791 [bsky, 7 points, 0 comments]
• Probably will be a bit of a tooling headache (was what stopped me from using CloudRM). Still thinking about the DeepSeek paper: arxiv.org/abs/2504.02495 [bsky, 1 points, 0 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling https://arxiv.org/abs/2504.02495 https://news.ycombinator.com/item?id=43578430 [bsky, 0 points, 0 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling https://arxiv.org/abs/2504.02495 [bsky, 0 points, 0 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling https://arxiv.org/abs/2504.02495 [comments] [105 points] [bsky, 0 points, 0 comments]
• 2. Inference-Time Scaling for Reward Modeling DeepSeek-GRM introduces SPCT training for scalable reward models. Uses meta-RMs and generative critiques arxiv.org/abs/2504.02495 [bsky, 0 points, 1 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling https://arxiv.org/abs/2504.02495 (https://news.ycombinator.com/item?id=43578430) [bsky, 0 points, 0 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• DeepSeek GRM arxiv.org/abs/2504.024... [bsky, 0 points, 0 comments]
• DeepSeek-GRM arxiv.org/abs/2504.02495 [bsky, 0 points, 2 comments]
• https://arxiv.org/abs/2504.02495 大規模言語モデル(LLM)における汎用報酬モデリングに関する研究。 推論時のスケーリング手法を提案し、性能向上を図る。 特に詳細な情報は論文を参照。 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2504.02495 이거 보다가 갑자기 드는 생각인데, 아니 이렇게 기술이 발전하고 맥락도 점점 더 잘 파악고, 창의적으로 발전하기 위한 정보들이 이렇게 많이 나오는데, 한국 챗봇 무능한 거 씁쓸했음… 질문에 해당하는 링크만 띄워주는 역할 할 거면 무슨 소용이에요…? [bsky, 0 points, 1 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling https://arxiv.org/abs/2504.02495 (https://news.ycombinator.com/item?id=43578430) [bsky, 0 points, 0 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling Article URL: https://arxiv.org/ab... https://arxiv.org/abs/2504.02495 Event Attributes [bsky, 0 points, 0 comments]
• DeepSeek: Inference-Time Scaling for Generalist Reward Modeling [bsky, 0 points, 0 comments]

Related

• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• Generalist Reward Models: Found Inside Large Language Models
• GRAM: A Generative Foundation Reward Model for Reward Generalization
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
• Reward Reasoning Model