A Comparative Study of Quality Evaluation Methods for Text Summarization

2024/06/30 by Huyen N. Nguyen, Haihua Chen, Nguyen, Huyen +5 · 7 citations
Computer Science · Decision Sciences · #Advanced Text Analysis Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2407.00747

openalex publication_date 2024/06/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Evaluating text summarization has been a challenging task in natural language processing (NLP). Automatic metrics which heavily rely on reference summaries are not suitable in many situations, while human evaluation is time-consuming and labor-intensive. To bridge this gap, this paper proposes a novel method based on large language models (LLMs) for evaluating text summarization. We also conducts a comparative study on eight automatic metrics, human evaluation, and our proposed LLM-based method. Seven different types of state-of-the-art (SOTA) summarization models were evaluated. We perform extensive experiments and analysis on datasets with patent documents. Our results show that LLMs evaluation aligns closely with human evaluation, while widely-used automatic metrics such as ROUGE-2, BERTScore, and SummaC do not and also lack consistency. Based on the empirical comparison, we propose a LLM-powered framework for automatically evaluating and improving text summarization, which is beneficial and could attract wide attention among the community.

Cited by

• DiscoVerse: Multi-Agent Pharmaceutical Co-Scientist for Traceable Drug Discovery and Reverse Translation
• Medical Interpretability and Knowledge Maps of Large Language Models
• Taming LLMs with Negative Samples: A Reference-Free Framework to Evaluate Presentation Content with Actionable Feedback
• Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System
• TF1-EN-3M: Three Million Synthetic Moral Fables for Training Small, Open Language Models
• MIDAS: Multi-LLM Iterative Data-Adaptive Summarization
• An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models

Related

• A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models
• Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Models
• An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models
• Evaluating LLMs and Pre-trained Models for Text Summarization Across Diverse Datasets
• Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization