LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation

2024/06/05 by Yi-Pei Chen, Chen, Yi-Pei, KuanChao Chu +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2406.02863

openalex publication_date 2024/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This research investigates the effect of prompt design on dialogue evaluation using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for dialogue evaluation remains challenging due to model sensitivity and subjectivity in dialogue assessments. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a "reason-first" approach yielding more comprehensive evaluations. This insight is crucial for enhancing the accuracy and consistency of LLM-based evaluations.

Cited by

• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations

Related

• Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation
• DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation
• Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialogues
• Evaluating Large Language Models at Evaluating Instruction Following
• Leveraging LLMs for Dialogue Quality Measurement