Weakly Supervised Grammatical Error Correction using Iterative Decoding

2018/10/31 by Jared Lichtarge, Christopher Alberti, Lichtarge, Jared +7
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1811.01710

openalex publication_date 2018/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We describe an approach to Grammatical Error Correction (GEC) that is effective at making use of models trained on large amounts of weakly supervised bitext. We train the Transformer sequence-to-sequence model on 4B tokens of Wikipedia revisions and employ an iterative decoding strategy that is tailored to the loosely-supervised nature of the Wikipedia training corpus. Finetuning on the Lang-8 corpus and ensembling yields an F0.5 of 58.3 on the CoNLL'14 benchmark and a GLEU of 62.4 on JFLEG. The combination of weakly supervised training and iterative decoding obtains an F0.5 of 48.2 on CoNLL'14 even without using any labeled GEC data.

Citations

• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study

Related

• Corpora Generation for Grammatical Error Correction
• Grammatical Error Correction as GAN-like Sequence Labeling
• Towards Minimal Supervision BERT-based Grammar Error Correction
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule
• Adapting LLMs for Minimal-edit Grammatical Error Correction