Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

2025/07/23 by Anisha Gunjal, Gunjal, Anisha, Anthony Wang +11 · 65 citations
Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Innovative Teaching and Learning Methods #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.17746

openalex publication_date 2025/07/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding. However, extending it to real-world reasoning tasks is challenging, as evaluation depends on nuanced, multi-criteria judgments rather than binary correctness. Instance-specific rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post-training remains underexplored. We introduce Rubrics as Rewards (RaR), an on-policy reinforcement learning method that extends RLVR beyond verifiable domains by using rubric-based feedback. Across both medical and science domains, we evaluate multiple strategies for aggregating rubric feedback into rewards. The best RaR variant achieves relative improvements of up to 31% on HealthBench and 7% on GPQA-Diamond over popular LLM-as-judge baselines that rely on direct Likert-based rewards. These results demonstrate that RaR-trained policies adapt well to diverse evaluation formats, performing strongly on both rubric-based and multiple-choice tasks. Moreover, we find that using rubrics as structured reward signals yields better alignment for smaller judges and reduces performance variance across judge scales.

Citations

• Checklists Are Better Than Reward Models For Aligning Language Models
• Configurable Preference Tuning with Rubric-Guided Synthetic Data
• QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
• ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
• Learning to Reason without External Rewards
• General-Reasoner: Advancing LLM Reasoning Across All Domains
• Reward Reasoning Model
• R3: Robust Rubric-Agnostic Reward Models
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
• HealthBench: Evaluating Large Language Models Towards Improved Human Health
• RM-R1: Reward Modeling as Reasoning
• Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
• Process Reward Models That Think
• Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
• Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learning
• Process Reinforcement through Implicit Rewards
• MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
• SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
• HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• GPT-4o System Card
• Improving Reward Models with Synthetic Critiques
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
• Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
• ODIN: Disentangled Reward Mitigates Hacking in RLHF
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• A Long Way to Go: Investigating Length Correlations in RLHF
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Let's Verify Step by Step
• The False Promise of Imitating Proprietary LLMs
• Training language models to follow instructions with human feedback
• OpenAI o1 System Card

Cited by

• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
• How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift
• Codifying the Judge: Scalable Evaluation via Program Distillation
• From Context to Skills: Can Language Models Learn from Context Skillfully?
• Step-DeepResearch Technical Report
• Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
• ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment
• Escaping the Verifier: Learning to Reason via Demonstrations
• Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints
• Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
• AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO
• CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
• Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognition
• Reward Models are Metrics in a Trench Coat
• SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
• Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO
• MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes
• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• OlaMind: Towards Human-Like and Hallucination-Safe Customer Service for Retrieval-Augmented Dialogue
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
• ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge
• ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
• Rethinking On-policy Optimization for Query Augmentation
• InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
• An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs
• AutoRubric-R1V: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning
• Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
• Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
• What Generative Search Engines Like and How to Optimize Web Content Cooperatively
• Can Tool-Integrated Reinforcement Learning Generalize Across Diverse Domains?
• Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise
• Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL
• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
• Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• Intelligent AI Delegation
• Online Rubrics Elicitation from Pairwise Comparisons
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search
• PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition
• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
• Flux-OPD: On-Policy Distillation with Evolving Contexts
• References Improve LLM Alignment in Non-Verifiable Domains
• OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generation
• Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning
• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
• Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
• Reinforcement Learning with Rubric Anchors
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System
• Are Today's LLMs Ready to Explain Well-Being Concepts?
• Goal Alignment in LLM-Based User Simulators for Conversational AI
• Decomposer: Learning to Decompile Symbolic Music to Programs
• A Primer in Post-Training Reasoning Data: What We Know About How It Works
• Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
• QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
• Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks

Related

• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• Reinforcement Learning with Rubric Anchors
• Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
• Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning
• Online Rubrics Elicitation from Pairwise Comparisons