Perplexity from PLM Is Unreliable for Evaluating Text Quality

2022/10/12 by Yequan Wang, Wang, Yequan, Jiawen Deng +5 · 4 citations
Computer Science · #Advanced Text Analysis Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2210.05892

openalex publication_date 2022/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, amounts of works utilize perplexity~(PPL) to evaluate the quality of the generated text. They suppose that if the value of PPL is smaller, the quality(i.e. fluency) of the text to be evaluated is better. However, we find that the PPL referee is unqualified and it cannot evaluate the generated text fairly for the following reasons: (i) The PPL of short text is larger than long text, which goes against common sense, (ii) The repeated text span could damage the performance of PPL, and (iii) The punctuation marks could affect the performance of PPL heavily. Experiments show that the PPL is unreliable for evaluating the quality of given text. Last, we discuss the key problems with evaluating text quality using language models.

Cited by

• Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care
• Security and Privacy Challenges of Large Language Models: A Survey
• ReDi: Rectified Discrete Flow
• Versatile Symbolic Music-for-Music Modeling via Function Alignment

Related

• Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?
• Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification
• Geometric Metrics and LLMs: What They Measure and When They Work
• Confidence, Not Perplexity: A Better Metric for the Creative Era of LLMs
• What is Wrong with Perplexity for Long-context Language Modeling?