Large Language Models are Inconsistent and Biased Evaluators

2024/05/02 by Rickard Stureborg, Stureborg, Rickard, Dimitris Alikaniotis +3 · 22 citations
Computer Science · #68T37 #68T50 (Primary) 68T01 #91F20 (Secondary) #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2 #I.2.7 #I.7 #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2405.01724

openalex publication_date 2024/05/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.

Cited by

• What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation
• A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models
• Prompt perturbation and fraction facilitation sometimes strengthen Large Language Model scores
• Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions
• Approximating Human Preferences Using a Multi-Judge Learned System
• TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
• Structure-Conditional Minimum Bayes Risk Decoding
• Vision-language models learn the geometry of human perceptual space
• Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
• Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models
• Trace Length is a Simple Uncertainty Signal in Reasoning Models
• If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models
• From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Education
• FAIRGAMER: Evaluating Biases in the Application of Large Language Models to Video Games
• Are Humans as Brittle as Large Language Models?
• Evaluating the Evaluators: Are readability metrics good measures of readability?
• The illusion of a perfect metric: Why evaluating AI's words is harder than it looks
• Feedback Indicators: The Alignment between Llama and a Teacher in Language Learning
• BIPOLAR: Polarization-based granular framework for LLM bias evaluation
• Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
• Counterfactual Evaluation for Blind Attack Detection in LLM-based Evaluation Systems
• CliCARE: Grounding Large Language Models in Clinical Guidelines for Decision Support over Longitudinal Cancer Electronic Health Records

Related

• Benchmarking Cognitive Biases in Large Language Models as Evaluators
• No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
• Style Over Substance: Evaluation Biases for Large Language Models
• An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• Evaluating Scoring Bias in LLM-as-a-Judge