2020/11/21 by Ondřej Dušek, Zdeněk Kasner, Dušek, Ondřej +1 · 6 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Software Engineering Research #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2011.10819
openalex publication_date 2020/11/21 · openalex created_date 2023/11/27 · openalex updated_date 2026/07/28
A major challenge in evaluating data-to-text (D2T) generation is measuring\nthe semantic accuracy of the generated text, i.e. checking if the output text\ncontains all and only facts supported by the input data. We propose a new\nmetric for evaluating the semantic accuracy of D2T generation based on a neural\nmodel pretrained for natural language inference (NLI). We use the NLI model to\ncheck textual entailment between the input data and the output text in both\ndirections, allowing us to reveal omissions or hallucinations. Input data are\nconverted to text for NLI using trivial templates. Our experiments on two\nrecent D2T datasets show that our metric can achieve high accuracy in\nidentifying erroneous system outputs.\n