2021/04/12 by Oleg Vasilyev, Vasilyev, Oleg, John Bohannon +1
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2104.05156
6 pages, 1 figure, 3 tables
arxiv created 2021/04/12 · openalex publication_date 2021/04/12 · arxiv updated 2021/04/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is designed to find and count all possible minute inconsistencies of the summary with respect to the source document. The proposed ESTIME, Estimator of Summary-to-Text Inconsistency by Mismatched Embeddings, correlates with expert scores in summary-level SummEval dataset stronger than other common evaluation measures not only in Consistency but also in Fluency. We also introduce a method of generating subtle factual errors in human summaries. We show that ESTIME is more sensitive to subtle errors than other common evaluation measures.