2020/10/09 by Gonçalo Mordido, Mordido, Gonçalo, Christoph Meinel +1 · 2 citations
Computer Science · Social Sciences · #Natural Language Processing Techniques #Topic Modeling #Wikis in Education and Collaboration
paper · pdf · doi:10.48550/arxiv.2010.04606
We propose a family of metrics to assess language generation derived from\npopulation estimation methods widely used in ecology. More specifically, we use\nmark-recapture and maximum-likelihood methods that have been applied over the\npast several decades to estimate the size of closed populations in the wild. We\npropose three novel metrics: ME_\Petersen and ME_\CAPTURE,\nwhich retrieve a single-valued assessment, and ME_\Schnabel which\nreturns a double-valued metric to assess the evaluation set in terms of quality\nand diversity, separately. In synthetic experiments, our family of methods is\nsensitive to drops in quality and diversity. Moreover, our methods show a\nhigher correlation to human evaluation than existing metrics on several\nchallenging tasks, namely unconditional language generation, machine\ntranslation, and text summarization.\n