Exploring the features used for summary evaluation by Human and GPT

2025/12/22 by Zahra Sadeghi, Sadeghi, Zahra, Evangelos Milios +3
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Sentiment Analysis and Opinion Mining #Topic Modeling

paper · doi:10.48550/arxiv.2512.19620

openalex publication_date 2025/12/22 · openalex created_date 2025/12/24 · openalex updated_date 2026/07/28

Abstract

Summary assessment involves evaluating how well a generated summary reflects the key ideas and meaning of the source text, requiring a deep understanding of the content. Large Language Models (LLMs) have been used to automate this process, acting as judges to evaluate summaries with respect to the original text. While previous research investigated the alignment between LLMs and Human responses, it is not yet well understood what properties or features are exploited by them when asked to evaluate based on a particular quality dimension, and there has not been much attention towards mapping between evaluation scores and metrics. In this paper, we address this issue and discover features aligned with Human and Generative Pre-trained Transformers (GPTs) responses by studying statistical and machine learning metrics. Furthermore, we show that instructing GPTs to employ metrics used by Human can improve their judgment and conforming them better with human responses.

Citations

• J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
• LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA
• Improve LLM-as-a-Judge Ability as a General Ability
• Self-rationalization improves LLM as a fine-grained judge
• Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability
• An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4
• Fusion-Eval: Integrating Assistant Evaluators with LLMs
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
• SMART: Sentences as Basic Units for Text Evaluation
• CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
• BARTScore: Evaluating Generated Text as Text Generation
• SummEval: Re-evaluating Summarization Evaluation
• Language Models are Few-Shot Learners
• BLEURT: Learning Robust Metrics for Text Generation
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
• BERTScore: Evaluating Text Generation with BERT
• BLEU is Not Suitable for the Evaluation of Text Simplification
• Robust Locally Weighted Regression and Smoothing Scatterplots

Related

• Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT
• Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks
• Towards Understanding the Robustness of LLM-based Evaluations under Perturbations
• Finding a Balanced Degree of Automation for Summary Evaluation
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation