Generative Judge for Evaluating Alignment

2023/10/09 by Junlong Li, Shichao Sun, Li, Junlong +9 · 37 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.05470

openalex publication_date 2023/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The rapid development of Large Language Models (LLMs) has substantially expanded the range of tasks they can address. In the field of Natural Language Processing (NLP), researchers have shifted their focus from conventional NLP tasks (e.g., sequence tagging and parsing) towards tasks that revolve around aligning with human needs (e.g., brainstorming and email writing). This shift in task distribution imposes new requirements on evaluating these aligned models regarding generality (i.e., assessing performance across diverse scenarios), flexibility (i.e., examining under different protocols), and interpretability (i.e., scrutinizing models with explanations). In this paper, we propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios and accommodates diverse evaluation protocols (e.g., pairwise response comparison and single-response evaluation) with well-structured natural language critiques. To demonstrate the efficacy of our approach, we construct a new testbed covering 58 different scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models, by a large margin. We also provide detailed analysis and case studies to further reveal the potential of our method and make a variety of resources public at https://github.com/GAIR-NLP/auto-j.

Cited by

• Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
• SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation
• Unifying points of interest taxonomies: mapping OpenStreetMap tags to the Foursquare category system
• Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAG
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• LLM-as-a-Verifier: A General-Purpose Verification Framework
• OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
• Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
• When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling
• Budget-aware Test-time Scaling via Discriminative Verification
• R-WoM: Retrieval-augmented World Model For Computer-use Agents
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
• SITCOM: Scaling Inference-Time COMpute for VLAs
• OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation
• Judging with Confidence: Calibrating Autoraters to Preference Distributions
• Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
• Reference-Free Rating of LLM Responses via Latent Information
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
• DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
• DSCC-HS: A Dynamic Self-Reinforcing Framework for Hallucination Suppression in Large Language Models
• SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgement
• Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization
• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression
• EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation
• Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
• Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward
• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
• Test-time Prompt Intervention
• Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple Judges
• MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis

Related

• Direct Judgement Preference Optimization
• On Evaluating LLM Alignment by Evaluating LLMs as Judges
• Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks
• Aligning Black-box Language Models with Human Judgments
• Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation