Real-World Summarization: When Evaluation Reaches Its Limits

2025/07/15 by Patrícia Schmidtová, Ondřej Dušek, Schmidtová, Patrícia +3
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.11508

openalex publication_date 2025/07/15 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

We examine evaluation of faithfulness to input data in the context of hotel highlights: brief LLM-generated summaries that capture unique features of accommodations. Through human evaluation campaigns involving categorical error assessment and span-level annotation, we compare traditional metrics, trainable methods, and LLM-as-a-judge approaches. Our findings reveal that simpler metrics like word overlap correlate surprisingly well with human judgments (Spearman correlation rank of 0.63), often outperforming more complex methods when applied to out-of-domain data. We further demonstrate that while LLMs can generate high-quality highlights, they prove unreliable for evaluation as they tend to severely under- or over-annotate. Our analysis of real-world business impacts shows incorrect and non-checkable information pose the greatest risks. We also highlight challenges in crowdsourced evaluations.

Citations

• LLMs as Span Annotators: A Comparative Study of LLMs and Humans
• A Survey on LLM-as-a-Judge
• Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
• Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
• Large Language Models Meet NLP: A Survey
• Proving Test Set Contamination in Black Box Language Models
• This is not correct! Negation-aware Evaluation of Language Generation Systems
• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
• Finetuned Language Models Are Zero-Shot Learners
• Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
• Language-agnostic BERT Sentence Embedding
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• BERTScore: Evaluating Text Generation with BERT

Related

• Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM
• UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs
• AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume
• LFOSum: Summarizing Long-form Opinions with Large Language Models
• UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches