Think-J: Learning to Think for Generative LLM-as-a-Judge

2025/05/20 by Huang Hui, Huang, Hui, Yancheng He +14 · 7 citations
Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Comparative and International Law Studies #Computation and Language (cs.CL) #FOS: Computer and information sciences #Legal Education and Practice Innovations

paper · pdf · doi:10.48550/arxiv.2505.14268

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

LLM-as-a-Judge refers to the automatic modeling of preferences for responses generated by Large Language Models (LLMs), which is of significant importance for both LLM evaluation and reward modeling. Although generative LLMs have made substantial progress in various tasks, their performance as LLM-Judge still falls short of expectations. In this work, we propose Think-J, which improves generative LLM-as-a-Judge by learning how to think. We first utilized a small amount of curated data to develop the model with initial judgment thinking capabilities. Subsequently, we optimize the judgment thinking traces based on reinforcement learning (RL). We propose two methods for judgment thinking optimization, based on offline and online RL, respectively. The offline method requires training a critic model to construct positive and negative examples for learning. The online method defines rule-based reward as feedback for optimization. Experimental results showed that our approach can significantly enhance the evaluation capability of generative LLM-Judge, surpassing both generative and classifier-based LLM-Judge without requiring extra human annotations.

Citations

• Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
• RM-R1: Reward Modeling as Reasoning
• Inference-Time Scaling for Generalist Reward Modeling
• JudgeLRM: Large Reasoning Models as a Judge
• AIR: Complex Instruction Generation via Automatic Iterative Refinement
• LIMO: Less is More for Reasoning
• A Survey on LLM-as-a-Judge
• Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives
• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution
• HelpSteer2-Preference: Complementing Ratings with Preferences
• The Llama 3 Herd of Models
• OffsetBias: Leveraging Debiased Data for Tuning Evaluators
• Improving Reward Models with Synthetic Critiques
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Secrets of RLHF in Large Language Models Part II: Reward Modeling
• CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• Generative Judge for Evaluating Alignment
• Statistical Rejection Sampling Improves Preference Optimization
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GPT-4 Technical Report
• Training language models to follow instructions with human feedback
• Proximal Policy Optimization Algorithms
• HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks

Cited by

• RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
• ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding
• The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization
• Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
• DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveys

Related

• Improve LLM-as-a-Judge Ability as a General Ability
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models