JudgeLRM: Large Reasoning Models as a Judge

2025/03/31 by Nuo Chen, Chen, Nuo, Zhiyuan Hu +11 · 44 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Explainable Artificial Intelligence (XAI) #Feature (linguistics) #Non-monotonic logic #Reinforcement learning #Scalability #Stability (learning theory) #Topic Modeling #Value (mathematics)

paper · pdf · doi:10.48550/arxiv.2504.00050

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/03/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.

Cited by

• Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation
• RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
• CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
• Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• Judging by the Rules: Compliance-Aligned Framework for Modern Slavery Statement Monitoring
• Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network
• Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
• OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• Think Twice: Branch-and-Rethink Reasoning Reward Model
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions
• Hearing Health in Home Healthcare: Leveraging LLMs for Illness Scoring and ALMs for Vocal Biomarker Extraction
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• mR3: Multilingual Rubric-Agnostic Reward Reasoning Models
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents
• ReviewRL: Towards Automated Scientific Review with RL
• Decomposed Reasoning with Reinforcement Learning for Relevance Assessment in UGC Platforms
• Libra: Assessing and Improving Reward Model by Learning to Think
• Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
• ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models
• Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict
• PulseReddit: A Novel Reddit Dataset for Benchmarking MAS in High-Frequency Cryptocurrency Trading
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
• RM-R1: Reward Modeling as Reasoning
• Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
• RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation
• Think-J: Learning to Think for Generative LLM-as-a-Judge
• Reward Reasoning Model
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
• R3: Robust Rubric-Agnostic Reward Models
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
• An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
• Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
• OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
• Reward Modeling for Scientific Writing Evaluation
• CHARM: Calibrating Reward Models With Chatbot Arena Scores

Related

• MR. Judge: Multimodal Reasoner as a Judge
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• Improve LLM-as-a-Judge Ability as a General Ability
• Assessing Judging Bias in Large Reasoning Models: An Empirical Study
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards