NoveltyBench: Evaluating Language Models for Humanlike Diversity

2025/04/07 by Yiming Zhang, Zhang, Yiming, Harshita Diddee +11 · 21 citations
Social Sciences · Medicine · Computer Science · #Ethics and Social Impacts of AI #Artificial Intelligence in Healthcare and Education #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.05228

Abstract

Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality.

Cited by

• Enhancing Diversity of LLM-Generated Educational Tasks
• Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories
• The One-Word Census: Answer-Choice Conformity Across 44 Language Models
• MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
• Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks
• Mode-Conditioning Unlocks Superior Test-Time Scaling
• Optimizing Diversity and Quality through Base-Aligned Model Collaboration
• G2: Guided Generation for Enhanced Output Diversity in LLMs
• CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge
• Magellan: Guided MCTS for Latent Space Exploration and Novelty Generation
• String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generation
• KL-Regularized Reinforcement Learning is Designed to Mode Collapse
• Cultural Alien Sampler: Open-ended art generation balancing originality and coherence
• What Shapes a Creative Machine Mind? Comprehensively Benchmarking Creativity in Foundation Models
• LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Search
• Polychromic Objectives for Reinforcement Learning
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• Avoidance Decoding for Diverse Multi-Branch Story Generation
• The Ramon Llull's Thinking Machine for Automated Ideation

Related

• Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models
• Growing a Tail: Increasing Output Diversity in Large Language Models
• Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting
• Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
• Benchmarking Linguistic Diversity of Large Language Models