Better Summarization Evaluation with Word Embeddings for ROUGE

2015/08/25 by Jun-Ping Ng, Ng, Jun-Ping, Viktoria Abrecht +1 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1508.06034

openalex publication_date 2015/08/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

ROUGE is a widely adopted, automatic evaluation measure for text summarization. While it has been shown to correlate well with human judgements, it is biased towards surface lexical similarities. This makes it unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing. We study the effectiveness of word embeddings to overcome this disadvantage of ROUGE. Specifically, instead of measuring lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our proposal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank coefficients.

Cited by

• Exploring question answering: metric analysis and evaluation framework for enhanced interpretability
• IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations
• Navigating Through Paper Flood: Advancing LLM-based Paper Evaluation through Domain-Aware Retrieval and Latent Reasoning

Related

• A Semantically Motivated Approach to Compute ROUGE Scores
• ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks
• Revisiting Summarization Evaluation for Scientific Articles
• ROUGE-K: Do Your Summaries Have Keywords?
• Leveraging Word Embeddings for Spoken Document Summarization