PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models

2024/11/22 by Gerald Friedland, Xin Huang, Friedland, Gerald +9 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #E.4 #FOS: Computer and information sciences #I.2.m #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2411.15320

openalex publication_date 2024/11/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose PPLqa, an easy to compute, language independent, information-theoretic metric to measure the quality of responses of generative Large Language Models (LLMs) in an unsupervised way, without requiring ground truth annotations or human supervision. The method and metric enables users to rank generative language models for quality of responses, so as to make a selection of the best model for a given task. Our single metric assesses LLMs with an approach that subsumes, but is not explicitly based on, coherence and fluency (quality of writing) and relevance and consistency (appropriateness of response) to the query. PPLqa performs as well as other related metrics, and works better with long-form Q&A. Thus, PPLqa enables bypassing the lengthy annotation process required for ground truth evaluations, and it also correlates well with human and LLM rankings.

Cited by

• Self-Aligned Reward: Towards Effective and Efficient Reasoners

Related

• Ranking Large Language Models without Ground Truth
• Toward a unified framework for data-efficient evaluation of large language models
• Language Models can Evaluate Themselves via Probability Discrepancy
• Are Your LLMs Capable of Stable Reasoning?
• Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks