Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge

2025/05/25 by Zhuo Liu, Moxin Li, Liu, Zhuo +7 · 2 citations
Business, Management and Accounting · Social Sciences · #Artificial Intelligence in Law #Computation and Language (cs.CL) #Dispute Resolution and Class Actions #FOS: Computer and information sciences #Legal Education and Practice Innovations

paper · pdf · doi:10.48550/arxiv.2505.19176

openalex publication_date 2025/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

LLM-as-a-Judge employs large language models (LLMs), such as GPT-4, to evaluate the quality of LLM-generated responses, gaining popularity for its cost-effectiveness and strong alignment with human evaluations. However, training proxy judge models using evaluation data generated by powerful teacher models introduces a critical yet previously overlooked issue: teacher preference bias, where the proxy judge model learns a biased preference for responses from the teacher model. To tackle this problem, we propose a novel setting that incorporates an additional assistant model, which is not biased toward the teacher model's responses, to complement the training data. Building on this setup, we introduce AGDe-Judge, a three-stage framework designed to debias from both the labels and feedbacks in the training data. Extensive experiments demonstrate that AGDe-Judge effectively reduces teacher preference bias while maintaining strong performance across six evaluation benchmarks. Code is available at https://github.com/Liuz233/AGDe-Judge.

Citations

• Do LLM Evaluators Prefer Themselves for a Reason?
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
• Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
• Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
• LLM Evaluators Recognize and Favor Their Own Generations
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Humans or LLMs as the Judge? A Study on Judgement Biases
• Debating with More Persuasive LLMs Leads to More Truthful Answers
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• Benchmarking Foundation Models with Language-Model-as-an-Examiner
• Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• UltraFeedback: Boosting Language Models with Scaled AI Feedback

Cited by

• Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling
• AGC-Bench: Measuring Artificial General Creativity

Related

• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
• Improve LLM-as-a-Judge Ability as a General Ability
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
• Quantitative LLM Judges
• Direct Judgement Preference Optimization