deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

2015/06/23 by Michel Galley, Galley, Michel, Chris Brockett +15 · 12 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1506.06863

6 pages, to appear at ACL 2015

openalex publication_date 2015/06/23 · arxiv created 2015/06/24 · arxiv updated 2015/06/25 · openalex created_date 2022/10/03 · openalex updated_date 2026/07/28

Abstract

We introduce Discriminative BLEU (deltaBLEU), a novel metric for intrinsic evaluation of generated text in tasks that admit a diverse range of possible outputs. Reference strings are scored for quality by human raters on a scale of [-1, +1] to weight multi-reference BLEU. In tasks involving generation of conversational responses, deltaBLEU correlates reasonably with human judgments and outperforms sentence-level and IBM BLEU in terms of both Spearman's rho and Kendall's tau.

Cited by

• Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA
• MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering
• Building Language Models for Text with Named Entities
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• An Attentional Neural Conversation Model with Improved Specificity
• Deep Reinforcement Learning for Dialogue Generation
• Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges
• Polite Dialogue Generation Without Parallel Data
• Neural Language Generation: Formulation, Methods, and Evaluation
• Learning to Disambiguate by Asking Discriminative Questions
• Calibrating Translation Decoding with Quality Estimation on LLMs
• Reinforcement Learning Based Emotional Editing Constraint Conversation Generation

Related

• NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
• BLEURT: Learning Robust Metrics for Text Generation
• CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation