Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA

2024/08/17 by Sher Badshah, Badshah, Sher, Hassan Sajjad +1 · 15 citations
Social Sciences · #68T07 #68T20 #68T50 #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Comparative and International Law Studies #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.0 #I.2.2 #I.2.7 #Legal Education and Practice Innovations

paper · pdf · doi:10.48550/arxiv.2408.09235

openalex publication_date 2024/08/17 · openalex created_date 2024/10/01 · openalex updated_date 2026/07/28

Abstract

The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics.

Cited by

• Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles
• M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models
• OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation
• What If Moderation Didn't Mean Suppression? A Case for Personalized Content Transformation
• CARGO: A Framework for Confidence-Aware Routing of Large Language Models
• Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
• Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluation
• Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches
• Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation
• Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat
• Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• Set-LLM: A Permutation-Invariant LLM
• STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories
• TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models

Related

• CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answering
• Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation
• TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models
• Evaluating Open-QA Evaluation
• MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs