Confidence, Not Perplexity: A Better Metric for the Creative Era of LLMs

2025/10/05 by Parupudi, V. S. Raghu
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.08596

Abstract

Reference-free metrics like self-perplexity are strongly biased against creative text generation. We propose the Confidence Score (CS), derived from a model's output probability distribution, as a less biased alternative. Experiments on gpt-4o-mini show that while fluency-based metrics prefer novel responses in 0% of cases on 99 creative prompts, our CS does so 19% of the time, a statistically significant difference (95% CI for difference: [11.1%, 27.3%]). We also show that CS effectively distinguishes between easy, medium, and hard tasks, confirmed by non-overlapping confidence intervals. The Confidence Score thus mitigates the creativity bias of traditional metrics while retaining their core evaluative strengths, offering a more balanced assessment for modern LLMs.

Citations

• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• SummEval: Re-evaluating Summarization Evaluation
• The Curious Case of Neural Text Degeneration
• BERTScore: Evaluating Text Generation with BERT

Related

• Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment
• CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?
• Gauging Overprecision in LLMs: An Empirical Study
• On Verbalized Confidence Scores for LLMs
• Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models