2019/10/10 by Ondřej Dušek, Dušek, Ondřej, Karin Sevegnani +5 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1910.04731
openalex publication_date 2019/10/10 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
We present a recurrent neural network based system for automatic quality\nestimation of natural language generation (NLG) outputs, which jointly learns\nto assign numerical ratings to individual outputs and to provide pairwise\nrankings of two different outputs. The latter is trained using pairwise hinge\nloss over scores from two copies of the rating network.\n We use learning to rank and synthetic data to improve the quality of ratings\nassigned by our system: we synthesise training pairs of distorted system\noutputs and train the system to rank the less distorted one higher. This leads\nto a 12% increase in correlation with human ratings over the previous\nbenchmark. We also establish the state of the art on the dataset of relative\nrankings from the E2E NLG Challenge (Du vsek et al., 2019), where synthetic\ndata lead to a 4% accuracy increase over the base model.\n