Using tournaments to calculate AUROC for zero-shot classification with LLMs

2025/02/20 by Yoon, WonJin, Bulovic, Ian, Miller, Timothy A. · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2502.15018

Abstract

Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.

Cited by

• AGENT-X: Adaptive Guideline-based Expert Network for Threshold-free AI-generated teXt detection

Related

• Learning to Identify Top Elo Ratings: A Dueling Bandits Approach
• Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat
• Investigating Non-Transitivity in LLM-as-a-Judge
• Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores
• Elo Uncovered: Robustness and Best Practices in Language Model Evaluation