Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling

2025/10/20 by Xie, Lipeng, Sen Huang, Huang, Sen +20 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.17314

openalex publication_date 2025/10/20 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/29

Abstract

Conventional reward modeling relies on gradient descent over neural weights, creating opaque, data-hungry "black boxes." We propose a paradigm shift from implicit to explicit reward parameterization, recasting optimization from continuous weight spaces to the discrete space of natural language rubrics. We introduce a training-free framework based on iterative rubric learning: it locally induces discriminative criteria via verification-driven refinement, and globally compresses the candidate criteria pool into a compact core set by maximizing an information-theoretic coding rate objective. We organize the compressed core set into a hierarchical rubric structure -- high-level evaluation dimensions supported by concrete verification checks -- serving as an interpretable, portable reward function. Empirically, our approach challenges prevailing data scaling assumptions: using only 70 preference pairs, our rubric-guided judges outperform fully trained reward models on diverse benchmarks. For instance, Qwen3-8B equipped with our learned rubrics achieves 80.91% on RewardBench2, surpassing the specialized Skywork-Reward-V2-Qwen3-8B (78.20%). These results demonstrate that alignment signals are highly compressible and can be effectively captured through explicit symbolic search.

Citations

• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• Checklists Are Better Than Reward Models For Aligning Language Models
• Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
• AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning
• RewardBench 2: Advancing Reward Model Evaluation
• Reward Reasoning Model
• R3: Robust Rubric-Agnostic Reward Models
• HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
• Qwen3 Technical Report
• HealthBench: Evaluating Large Language Models Towards Improved Human Health
• RM-R1: Reward Modeling as Reasoning
• Inference-Time Scaling for Generalist Reward Modeling
• Rule Based Rewards for Language Model Safety
• CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• Inverse Constitutional AI: Compressing Preferences into Principles
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• RewardBench: Evaluating Reward Models for Language Modeling
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• Generative Judge for Evaluating Alignment
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Large Language Models are not Fair Evaluators
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Emergent Analogical Reasoning in Large Language Models
• Affective Coherence Monitoring for Transformer-Based Language Models
• Scaling Laws for Reward Model Overoptimization
• Training language models to follow instructions with human feedback
• Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
• Deep reinforcement learning from human preferences
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Cited by

• ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment
• Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints

Related

• R3: Robust Rubric-Agnostic Reward Models
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• RewardBench: Evaluating Reward Models for Language Modeling
• RewardAnything: Generalizable Principle-Following Reward Models
• A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future