Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization

2025/05/15 by Ximing Dong, Dong, Ximing, Shaowei Wang +5 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.10736

openalex publication_date 2025/05/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Optimizing Large Language Model (LLM) performance requires well-crafted prompts, but manual prompt engineering is labor-intensive and often ineffective. Automated prompt optimization techniques address this challenge but the majority of them rely on randomly selected evaluation subsets, which fail to represent the full dataset, leading to unreliable evaluations and suboptimal prompts. Existing coreset selection methods, designed for LLM benchmarking, are unsuitable for prompt optimization due to challenges in clustering similar samples, high data collection costs, and the unavailability of performance data for new or private datasets. To overcome these issues, we propose IPOMP, an Iterative evaluation data selection for effective Prompt Optimization using real-time Model Performance. IPOMP is a two-stage approach that selects representative and diverse samples using semantic clustering and boundary analysis, followed by iterative refinement with real-time model performance data to replace redundant samples. Evaluations on the BIG-bench dataset show that IPOMP improves effectiveness by 1.6% to 5.3% and stability by at least 57% compared with SOTA baselines, with minimal computational overhead below 1%. Furthermore, the results demonstrate that our real-time performance-guided refinement approach can be universally applied to enhance existing coreset selection methods.

Citations

• 100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances
• Task Facet Learning: A Structured Approach to Prompt Optimization
• Efficient Universal Goal Hijacking with Semantics-guided Prompt Organization
• A Survey on Data Selection for Language Models
• tinyBenchmarks: evaluating LLMs with fewer examples
• Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models
• Anchor Points: Benchmarking Models with Much Fewer Examples
• Optimal Sample Selection Through Uncertainty Estimation and Its Application in Deep Learning
• Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• Large Language Models Are Human-Level Prompt Engineers
• RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
• Large Language Models are Zero-Shot Reasoners
• Active Learning by Acquiring Contrastive Examples
• Just Train Twice: Improving Group Robustness without Training Group Information
• Contextual Diversity for Active Learning
• Confident Coreset for Active Learning in Medical Image Analysis
• Small-GAN: Speeding Up GAN Training Using Core-sets
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Selection via Proxy: Efficient Data Selection for Deep Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Adversarial Active Learning for Deep Networks: a Margin Based Approach
• "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• Super-Samples from Kernel Herding
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers

Cited by

• MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization
• APEX: Automated Prompt Engineering eXpert with Dynamic Data Selection

Related

• Speculative Coreset Selection for Task-Specific Fine-tuning
• Towards Optimizing with Large Language Models
• Crafting Efficient Fine-Tuning Strategies for Large Language Models
• Dual-Phase Accelerated Prompt Optimization
• Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models