Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts

2025/04/29 by Hanhua Hong, Hong, Hanhua, Chenghao Xiao +9 · 1 voice · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2504.21117

openalex publication_date 2025/04/29 · arxiv published 2025/04/29 · arxiv updated 2025/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Evaluating natural language generation systems is challenging due to the diversity of valid outputs. While human evaluation is the gold standard, it suffers from inconsistencies, lack of standardisation, and demographic biases, limiting reproducibility. LLM-based evaluators offer a scalable alternative but are highly sensitive to prompt design, where small variations can lead to significant discrepancies. In this work, we propose an inversion learning method that learns effective reverse mappings from model outputs back to their input instructions, enabling the automatic generation of highly effective, model-specific evaluation prompts. Our method requires only a single evaluation sample and eliminates the need for time-consuming manual prompt engineering, thereby improving both efficiency and robustness. Our work contributes toward a new direction for more robust and efficient LLM-based evaluation.

Citations

• Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models
• A Systematic Survey of Automatic Prompt Optimization Techniques
• HPSS: Heuristic Prompting Strategy Search for LLM Evaluators
• InfiR : Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning
• Idiosyncrasies in Large Language Models
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation
• What do Large Language Models Need for Machine Translation Evaluation?
• The Llama 3 Herd of Models
• PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation
• Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics
• What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering
• On the Worst Prompt Performance of Large Language Models
• ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models
• Efficient multi-prompt evaluation of LLMs
• SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation
• DAGER: Exact Gradient Inversion for Large Language Models
• Extracting Prompts by Inverting LLM Outputs
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Language Model Inversion
• LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
• The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
• Text Embeddings Reveal (Almost) As Much As Text
• Large Language Models as Optimizers
• AutoHint: Automatic Prompt Optimization with Hint Generation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• A Survey of Large Language Models
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• Towards a Unified Multi-Dimensional Evaluator for Text Generation
• Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
• BARTScore: Evaluating Generated Text as Text Generation
• LoRA: Low-Rank Adaptation of Large Language Models
• Extracting Training Data from Large Language Models
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• SummEval: Re-evaluating Summarization Evaluation
• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
• Information Leakage in Embedding Models
• Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
• The Curious Case of Neural Text Degeneration
• BERTScore: Evaluating Text Generation with BERT
• Path-Specific Counterfactual Fairness

Cited by

• Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth

Discussions

• https://arxiv.org/abs/2504.21117 #cs.CL Result Details [bsky, 0 points, 0 comments]

Related

• HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation
• Automatic Evaluation of Generative Models with Instruction Tuning
• LLM-based NLG Evaluation: Current Status and Challenges
• Calibrating LLM-Based Evaluator
• Large Language Models Are Active Critics in NLG Evaluation