Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores

2025/10/16 by Robertson, Zachary
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2510.14966

Abstract

Pairwise comparisons of large language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions. Maximum-likelihood approaches to IRT use logistic links, but we find empirically that these transformations introduce curvature that breaks additivity: across three domains, the identity link yields median curl on raw data of 0.080-0.150 (P95 = [0.474, 0.580]), whereas probit/logit introduce substantially higher violations (median [0.245, 0.588], P95 [0.825, 2.252]). We derive this clipped-linear model from Gini entropy maximization, yielding a box-constrained least-squares formulation that handles boundary saturation. At 33% coverage, we achieve holdout RMSE 0.117 ± 0.008 while preserving agent rankings (Spearman ρ= 0.972 ± 0.015), three times fewer evaluations than full dense. Judge robustness analysis (GPT-4o-mini vs. Llama3-70b) shows strong agreement in agent rankings (ρ= 0.872) and consistent identity-link advantage. TVD-MI's geometry is best preserved by identity mapping for efficient LLM evaluation, applicable to other bounded-response domains.

Citations

• Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory
• Benchmarking LLMs' Judgments with No Gold Standard
• Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
• Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Crowdsourced Judgement Elicitation with Endogenous Proficiency
• On integral probability metrics, ϕ-divergences and binary classification
• Exact Matrix Completion via Convex Optimization
• Let's Measure Information Step-by-Step: LLM-Based Evaluation Beyond Vibes
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

Related

• Reference-Free Rating of LLM Responses via Latent Information
• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
• Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks
• JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
• Bridging Human and LLM Judgments: Understanding and Narrowing the Gap