Evaluation of Text Generation: A Survey

2020/06/26 by Aslı Çelikyılmaz, Asli Celikyilmaz, Elizabeth Clark +4 · 1 voice · 200 citations
Computer Science · Engineering · #Artificial intelligence #Automatic summarization #Computer science #Data science #Engineering #Focus (optics) #Information retrieval #Machine learning #Natural Language Processing Techniques #Natural language #Natural language generation #Natural language processing #Speech and dialogue systems #Systems engineering #Task (project management) #Text generation #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2006.14799

published in arXiv (Cornell University) (Cornell University) · 47 pages (revised version)

openalex publication_date 2020/06/26 · arxiv created 2021/05/18 · arxiv updated 2021/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/03

Abstract

The paper surveys evaluation methods of natural language generation (NLG) systems that have been developed in the last few years. We group NLG evaluation methods into three categories: (1) human-centric evaluation metrics, (2) automatic metrics that require no training, and (3) machine-learned metrics. For each category, we discuss the progress that has been made and the challenges still being faced, with a focus on the evaluation of recently proposed NLG tasks and neural NLG models. We then present two examples for task-specific NLG evaluations for automatic text summarization and long text generation, and conclude the paper by proposing future research directions.

Citations

• On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.
• A Coefficient of Agreement for Nominal Scales
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Few-shot Natural Language Generation for Task-Oriented Dialog
• On the Dangers of Stochastic Parrots

Cited by

• grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP
• Is GPT-OSS All You Need? Benchmarking Large Language Models for Financial Intelligence and the Surprising Efficiency Paradox
• OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
• Do Encoder Representations of Generative Dialogue Models Encode Sufficient Information about the Task ?
• Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
• LLM-as-a-Judge for Evaluating System Responses in Conversational Music Recommendation
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• Q²: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
• A Layered Intuition -- Method Model with Scope Extension for LLM Reasoning
• Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions
• Online Back-Parsing for AMR-to-Text Generation
• Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth
• The illusion of a perfect metric: Why evaluating AI's words is harder than it looks
• A Conditional Generative Matching Model for Multi-lingual Reply Suggestion
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• On the interaction of automatic evaluation and task framing in headline style transfer
• COGENT: A Curriculum-oriented Framework for Generating Grade-appropriate Educational Content
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey
• Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task
• The Thin Line Between Comprehension and Persuasion in LLMs
• Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generation
• Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models
• Statistical Multicriteria Evaluation of LLM-Generated Text
• From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentary
• References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
• Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
• The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing
• Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion
• Neural Language Generation: Formulation, Methods, and Evaluation
• Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
• An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
• Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest
• The Ultimate Cookbook for Invisible Poison: Crafting Subtle Clean-Label Text Backdoors with Style Attributes
• Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation
• CPR: Leveraging LLMs for Topic and Phrase Suggestion to Facilitate Comprehensive Product Reviews
• LLMs as Span Annotators: A Comparative Study of LLMs and Humans

Discussions

• Evaluation of Text Generation: A Survey [hn, 46 points, 0 comments]

Related

• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
• Reference-free Evaluation Metrics for Text Generation: A Survey
• LLM-based NLG Evaluation: Current Status and Challenges
• LLM-based NLG Evaluation: Current Status and Challenges
• Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation