2017/08/05 by Ondřej Dušek, Dušek, Ondřej, Jekaterina Novikova +3 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.1708.01759
Accepted as a regular paper to 1st Workshop on Learning to Generate Natural Language (LGNL), Sydney, 10 August 2017
arxiv created 2017/08/05 · openalex publication_date 2017/08/05 · arxiv updated 2017/08/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.