Evaluating the Diversity and Quality of LLM Generated Content

2025/04/16 by Alexander Shypula, Shypula, Alexander, Shuo Li +9 · 54 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Mobile Crowdsensing and Crowdsourcing #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.12522

openalex publication_date 2025/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work suggests that preference-tuning techniques -- such as Reinforcement Learning from Human Feedback (RLHF) methods like PPO and GRPO, as well as alternatives like DPO -- reduce diversity, creating a dilemma given that these models are widely deployed in applications requiring varied outputs. We argue that diversity without consideration of quality has limited practical value. To address this issue, we introduce a framework for measuring effective semantic diversity -- diversity among outputs that meet quality thresholds -- which better reflects the practical utility of large language models (LLMs). Using open-ended tasks that require no human intervention, we find counterintuitive results: when using diversity metrics that do not explicitly consider quality, preference-tuned models -- particularly those trained via RL -- often produce outputs with lower diversity; however, these same preference-tuned models generate greater effective semantic diversity than supervised fine-tuned (SFT) or base models. Our analysis further shows another trend: while larger models may exhibit greater effective semantic diversity than smaller models, the smaller models are consistently more parameter-efficient at producing unique content within a fixed sampling budget. These findings have practical implications for applications that require diverse yet high-quality outputs, from creative assistance to synthetic data generation.

Citations

• Diverse Preference Optimization
• Qwen2.5 Technical Report
• Benchmarking Linguistic Diversity of Large Language Models
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity
• On the Diversity of Synthetic Data and its Impact on Training Large Language Models
• Qwen2.5-Coder Technical Report
• The Llama 3 Herd of Models
• Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• Does Writing with Language Models Reduce Content Diversity?
• Code Llama: Open Foundation Models for Code
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data
• ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• ICE-Score: Instructing Large Language Models to Evaluate Code
• Whose Opinions Do Language Models Reflect?
• GPT-4 Technical Report
• Learning Performance-Improving Code Edits
• CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
• RL with KL penalties is better viewed as Bayesian inference
• Semantic Diversity in Dialogue with Natural Language Inference
• Training language models to follow instructions with human feedback
• Competition-Level Code Generation with AlphaCode
• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN
• CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
• Learning to summarize from human feedback
• Trading Off Diversity and Quality in Natural Language Generation
• Evaluating the Evaluation of Diversity in Natural Language Generation
• Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Fine-Tuning Language Models from Human Preferences
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• The Curious Case of Neural Text Degeneration
• BERTScore: Evaluating Text Generation with BERT
• Hierarchical Neural Story Generation
• Texygen: A Benchmarking Platform for Text Generation Models
• Proximal Policy Optimization Algorithms
• A Diversity-Promoting Objective Function for Neural Conversation Models
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• WeMusic-Agent: Efficient Conversational Music Recommendation via Knowledge Internalization and Agentic Boundary Learning
• CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models
• Personalized Reward Modeling for Text-to-Image Generation
• RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
• KL-Regularized Reinforcement Learning is Designed to Mode Collapse
• Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations
• Post-training Large Language Models for Diverse High-Quality Responses
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
• Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles
• Multimodal Mathematical Reasoning with Diverse Solving Perspective
• A Fast, Reliable, and Secure Programming Language for LLM Agents with Code Actions
• Effective Reinforcement Learning for Reasoning in Language Models
• Creative Preference Optimization
• Pairwise Calibrated Rewards for Pluralistic Alignment
• Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models
• Do Large Language Models Always Tell The Same Stories?
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
• Measuring LLM Novelty As The Frontier Of Original And High-Quality Output

Related

• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
• Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data
• Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models