JudgeLRM: Large Reasoning Models as a Judge

2025/03/31 by Nuo Chen, Chen, Nuo, Zhiyuan Hu +11 · 25 citations
Computer Science · Medicine · #Topic Modeling #Artificial Intelligence in Healthcare and Education #Explainable Artificial Intelligence (XAI)

paper · pdf · doi:10.48550/arxiv.2504.00050

Abstract

Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.

Cited by

• Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation
• RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
• CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
• Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• Judging by the Rules: Compliance-Aligned Framework for Modern Slavery Statement Monitoring
• Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network
• Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
• OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• Think Twice: Branch-and-Rethink Reasoning Reward Model
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions
• Hearing Health in Home Healthcare: Leveraging LLMs for Illness Scoring and ALMs for Vocal Biomarker Extraction
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• mR3: Multilingual Rubric-Agnostic Reward Reasoning Models
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents
• ReviewRL: Towards Automated Scientific Review with RL
• Decomposed Reasoning with Reinforcement Learning for Relevance Assessment in UGC Platforms
• Libra: Assessing and Improving Reward Model by Learning to Think

Related

• MR. Judge: Multimodal Reasoner as a Judge
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• Improve LLM-as-a-Judge Ability as a General Ability
• Assessing Judging Bias in Large Reasoning Models: An Empirical Study
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards