Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation

2022/08/24 by Cyril Chhun, Pierre Colombo, Chhun, Cyril +5 · 14 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2208.11646

43 pages, 38 figures. Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022)

openalex publication_date 2022/08/24 · openalex created_date 2022/08/27 · arxiv created 2022/09/15 · arxiv updated 2022/09/16 · openalex updated_date 2026/07/28

Abstract

Research on Automatic Story Generation (ASG) relies heavily on human and automatic evaluation. However, there is no consensus on which human evaluation criteria to use, and no analysis of how well automatic criteria correlate with them. In this paper, we propose to re-evaluate ASG evaluation. We introduce a set of 6 orthogonal and comprehensive human criteria, carefully motivated by the social sciences literature. We also present HANNA, an annotated dataset of 1,056 stories produced by 10 different ASG systems. HANNA allows us to quantitatively evaluate the correlations of 72 automatic metrics with human criteria. Our analysis highlights the weaknesses of current metrics for ASG and allows us to formulate practical recommendations for ASG evaluation.

Citations

• Efficient Estimation of Word Representations in Vector Space
• The Measurement of Observer Agreement for Categorical Data
• BERTScore: Evaluating Text Generation with BERT
• From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Transformers: State-of-the-Art Natural Language Processing
• Narrative Interpolation for Generating and Understanding Stories
• Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
• BARTScore: Evaluating Generated Text as Text Generation
• What are the best systems? New perspectives on NLP Benchmarking
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics

Cited by

• What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation
• AgentEval: Generative Agents as Reliable Proxies for Human Evaluation of AI-Generated Content
• Lightweight Latent Reasoning for Narrative Tasks
• Large Language Models as Search Engines: Societal Challenges
• Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
• CreAgentive: An Agent Workflow Driven Multi-Category Creative Generation Engine
• Once Upon a Time: Interactive Learning for Storytelling with Small Language Models
• PREFINE: Personalized Story Generation via Simulated User Critics and User-Specific Rubric Generation
• Re:Verse -- Can Your VLM Read a Manga?
• Can LLMs Generate Good Stories? Insights and Challenges from a Narrative Planning Perspective
• FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems
• StoryWriter: A Multi-Agent Framework for Long Story Generation
• The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing
• Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge

Related

• What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation
• HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation
• The Authenticity Gap in Human Evaluation
• OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics