JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

2026/05/24 by Russell Yang, Ruishi Chen, Pierce Kelaita +6 · 1 voice
Computer Science · Social Sciences · #Artificial Intelligence in Law #Benchmark (surveying) #Domain (mathematical analysis) #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #Inter-rater reliability #Pairwise comparison #Preference #Quality (philosophy) #Rank (graph theory) #Rubric #cs.AI #cs.CL #cs.CY

paper · pdf · open access · doi:10.48550/arxiv.2605.25240

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/05/24 · arxiv published 2026/05/24 · openalex created_date 2026/05/27 · arxiv updated 2026/06/04 · openalex updated_date 2026/07/28

Abstract

Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts on the same items. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, estimated difference = 0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.

Citations

• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
• ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• Online Rubrics Elicitation from Pairwise Comparisons
• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
• Reinforcement Learning with Rubric Anchors
• Checklists Are Better Than Reward Models For Aligning Language Models
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• Configurable Preference Tuning with Rubric-Guided Synthetic Data
• QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
• PaperBench: Evaluating AI's Ability to Replicate AI Research
• BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• Comparing Criteria Development Across Domain Experts, Lay Users, and Models in Large Language Model Evaluation
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• Humans or LLMs as the Judge? A Study on Judgement Biases
• Self-Rewarding Language Models
• Using large language models to promote health equity
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Aligning Large Language Models through Synthetic Feedback
• Large Language Models Encode Clinical Knowledge
• Large language models encode clinical knowledge
• Training language models to follow instructions with human feedback
• LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
• All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated\n Text
• When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
• When does pretraining help?
• A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering
• Annotation Artifacts in Natural Language Inference Data
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• A law of comparative judgment.

Discussions

• JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment arxiv.org/abs/2605.25240 [bsky, 0 points, 0 comments]

Related

• Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
• Are We on the Right Way to Assessing LLM-as-a-Judge?
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• LLM-as-a-Judge: Rapid Evaluation of Legal Document Recommendation for Retrieval-Augmented Generation
• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning