Reference-less Measure of Faithfulness for Grammatical Error Correction

2018/04/11 by Leshem Choshen, Omri Abend, Choshen, Leshem +1
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1804.03824

openalex publication_date 2018/04/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USim operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USim, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USim similarity score to the source; and (3) invalid corrections obtain a lower score.

Citations

• On the Variability of Interlanguage Systems*

Related

• There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction
• CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction
• CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction
• GLEU Without Tuning
• JELV: A Judge of Edit-Level Validity for Evaluation and Automated Reference Expansion in Grammatical Error Correction