Underreporting of errors in NLG output, and what to do about it

2021/08/02 by Emiel van Miltenburg, Miruna Clinciu, van Miltenburg, Emiel +19 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · doi:10.48550/arxiv.2108.01182

openalex publication_date 2021/08/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

Cited by

• LLM-as-a-qualitative-judge: automating error analysis in natural language generation

Related

• LLM-as-a-qualitative-judge: automating error analysis in natural language generation
• Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
• Contrastive Error Attribution for Finetuned Language Models
• Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices
• Evaluating NLG systems: A brief introduction