Rethinking and Refining the Distinct Metric

2022/02/28 by Siyang Liu, Liu, Siyang, Sahand Sabour +9 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Speech and dialogue systems #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2202.13587

4 pages, to be published at ACL2022

openalex publication_date 2022/02/28 · arxiv created 2022/04/03 · arxiv updated 2022/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Distinct-n score\citeLi2016 is a widely used automatic metric for evaluating diversity in language generation tasks. However, we observed that the original approach for calculating distinct scores has evident biases that tend to assign higher penalties to longer sequences. We refine the calculation of distinct scores by scaling the number of distinct tokens based on their expectations. We provide both empirical and theoretical evidence to show that our method effectively removes the biases existing in the original distinct score. Our experiments show that our proposed metric, Expectation-Adjusted Distinct (EAD), correlates better with human judgment in evaluating response diversity. To foster future research, we provide an example implementation at \urlhttps://github.com/lsy641/Expectation-Adjusted-Distinct.

Cited by

• Optimizing Diversity and Quality through Base-Aligned Model Collaboration
• G2: Guided Generation for Enhanced Output Diversity in LLMs
• KL-Regularized Reinforcement Learning is Designed to Mode Collapse
• Following the TRACE: A Structured Path to Empathetic Response Generation with Multi-Agent Models
• Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
• MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
• Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks
• LLM Probability Concentration: How Alignment Shrinks the Generative Horizon

Related

• Evaluating the Evaluation of Diversity in Natural Language Generation
• Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
• Measuring Diversity in Synthetic Datasets
• Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data