LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models

2024/02/16 by Minsuk Kahng, Kahng, Minsuk, Ian Tenney +17 · 14 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2402.10524

openalex publication_date 2024/02/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from this evaluation approach raises scalability and interpretability challenges. In this paper, we present LLM Comparator, a novel visual analytics tool for interactively analyzing results from automatic side-by-side evaluation. The tool supports interactive workflows for users to understand when and why a model performs better or worse than a baseline model, and how the responses from two models are qualitatively different. We iteratively designed and developed the tool by closely working with researchers and engineers at a large technology company. This paper details the user challenges we identified, the design and development of the tool, and an observational study with participants who regularly evaluate their models.

Cited by

• Uncovering Competency Gaps in Large Language Models and Their Benchmarks
• Evalet: Evaluating Large Language Models through Functional Fragmentation
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancement
• RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
• DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition
• ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Making
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models
• A Design Space for the Critical Validation of LLM-Generated Tabular Data
• Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations
• Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation
• An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
• Zero-shot adaptable task planning for autonomous construction robots: a comparative study of lightweight single and multi-AI agent systems

Related

• EasyJudge: an Easy-to-use Tool for Comprehensive Response Evaluation of LLMs
• State of What Art? A Call for Multi-Prompt LLM Evaluation
• LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Models
• Enhancing LLM Evaluations: The Garbling Trick
• LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition