LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization

2025/10/14 by Wu, Yuanchen, Verma, Saurabh, Lee, Justin +6
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2510.13907

Abstract

Large language models (LLMs) are highly sensitive to their input prompts, making prompt design a central challenge. While automatic prompt optimization (APO) reduces manual engineering, most approaches assume access to ground-truth references such as labeled validation data. In practice, however, collecting high-quality labels is costly and slow. We propose the Prompt Duel Optimizer (PDO), a sample-efficient framework for label-free prompt optimization. PDO formulates the problem as a dueling-bandit setting, where supervision signal comes from pairwise preference feedback provided by an LLM judge. The framework combines Double Thompson Sampling (D-TS), which prioritizes informative prompt comparisons, with Top-Performer Guided Mutation, which expands the candidate pool by mutating high-performing prompts. PDO naturally operates in label-free settings and can also incorporate partial labels to mitigate judge noise. Experiments on BIG-bench Hard (BBH) and MS MARCO show that PDO consistently outperforms baseline methods. Ablation studies further demonstrate the effectiveness of both D-TS and prompt mutation.

Citations

• Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers
• Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models
• A Systematic Survey of Automatic Prompt Optimization Techniques
• Self-Supervised Prompt Optimization
• A Survey on LLM-as-a-Judge
• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
• Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
• Efficient Prompt Optimization Through the Lens of Best Arm Identification
• Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
• Large Language Models as Optimizers
• Evaluating the Zero-shot Robustness of Instruction-tuned Language Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• Self-Refine: Iterative Refinement with Self-Feedback
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Large Language Models Are Human-Level Prompt Engineers
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Multi-dueling Bandits with Dependent Arms
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• Double Thompson Sampling for Dueling Bandits
• Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problem
• Multi-Armed Bandits in Metric Spaces

Related

• Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
• Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers
• Are Large Language Models Good Prompt Optimizers?
• Self-Supervised Prompt Optimization
• PLHF: Prompt Optimization with Few-Shot Human Feedback