ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering

2025/07/01 by Alexander Hoyle, Hoyle, Alexander, Lorena Calvo-Bartolomé +5 · 1 voice
#cs.CL

paper · pdf · doi:10.48550/arxiv.2507.00828

Abstract

Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations. Package, web interface, and data are at https://github.com/ahoho/proxann

Citations

• The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• A NEW MEASURE OF RANK CORRELATION

Discussions

• LLMs are often used for text annotation, especially in social science. In some cases, this involves placing text items on a scale: eg, 1 for liberal and 9 for conservative There are a few ways to acco [bsky, 27 points, 1 comments]

Related

• Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluation
• Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models
• Improving the TENOR of Labeling: Re-evaluating Topic Models for Content Analysis
• Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models
• The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?