Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

2024/01/13 by Zhen Li, Li, Zhen, Xiaohan Xu +10 · 13 citations
Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #Wikis in Education and Collaboration

paper · pdf · doi:10.48550/arxiv.2401.07103

openalex publication_date 2024/01/13 · openalex created_date 2024/01/18 · openalex updated_date 2026/07/28

Abstract

In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This paper aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.

Cited by

• Learning from Self Critique and Refinement for Faithful LLM Summarization
• Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
• OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews
• Expert Preference-based Evaluation of Automated Related Work Generation
• Establishing Best Practices for Building Rigorous Agentic Benchmarks
• A Conceptual Framework for AI Capability Evaluations
• References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
• ProRefine: Inference-Time Prompt Refinement with Textual Feedback
• RewardAnything: Generalizable Principle-Following Reward Models
• Ask, Retrieve, Summarize: A Modular Pipeline for Scientific Literature Summarization
• AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation
• From Physician Expertise to Clinical Agents: Preserving, Standardizing, and Scaling Physicians' Medical Expertise with Lightweight LLM
• Reward Modeling for Scientific Writing Evaluation

Related

• LLM-based NLG Evaluation: Current Status and Challenges
• LLM-based NLG Evaluation: Current Status and Challenges
• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
• Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability
• Large Language Models Are Active Critics in NLG Evaluation