deltaBLEU: A Discriminative Metric for Generation Tasks with\n Intrinsically Diverse Targets

2015/06/23 by Michel Galley, Galley, Michel, Chris Brockett +15 · 9 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1506.06863

openalex publication_date 2015/06/23 · openalex created_date 2022/10/03 · openalex updated_date 2026/07/28

Abstract

We introduce Discriminative BLEU (deltaBLEU), a novel metric for intrinsic\nevaluation of generated text in tasks that admit a diverse range of possible\noutputs. Reference strings are scored for quality by human raters on a scale of\n[-1, +1] to weight multi-reference BLEU. In tasks involving generation of\nconversational responses, deltaBLEU correlates reasonably with human judgments\nand outperforms sentence-level and IBM BLEU in terms of both Spearman's rho and\nKendall's tau.\n

Cited by

• Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
• MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering
• Building Language Models for Text with Named Entities
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• An Attentional Neural Conversation Model with Improved Specificity
• Deep Reinforcement Learning for Dialogue Generation
• Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges
• Polite Dialogue Generation Without Parallel Data
• Neural Language Generation: Formulation, Methods, and Evaluation

Related

• NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
• BLEURT: Learning Robust Metrics for Text Generation
• CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation