Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability

2024/06/26 by Xinyu Hu, Hu, Xinyu, Lin Li +7 · 13 citations
Computer Science · #AI-based Problem Solving and Planning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Fuzzy Logic and Control Systems #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2406.18365

openalex publication_date 2024/06/26 · openalex created_date 2024/06/28 · openalex updated_date 2026/07/28

Abstract

The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and model-based metrics. However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility. Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus NLG-Eval with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field. Furthermore, we propose Themis, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods. Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.

Cited by

• Exploring the features used for summary evaluation by Human and GPT
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• LLM-as-a-Verifier: A General-Purpose Verification Framework
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• The Oracle Has Spoken: A Multi-Aspect Evaluation of Dialogue in Pythia
• EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation
• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
• Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
• Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
• xVerify: Efficient Answer Verifier for Reasoning Model Evaluations

Related

• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges
• LLM-based NLG Evaluation: Current Status and Challenges
• LLM-based NLG Evaluation: Current Status and Challenges
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Large Language Models Are Active Critics in NLG Evaluation