2021/09/16 by Spencer Braun, Oleg Vasilyev, Braun, Spencer +5
Computer Science · #Computation and Language (cs.CL) #Data Mining Algorithms and Applications #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2109.08129
9 pages, 6 figures, 1 table, 3 appendixes
openalex publication_date 2021/09/16 · arxiv created 2021/12/08 · arxiv updated 2021/12/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.