S2LPP: Small-to-Large Prompt Prediction across LLMs

2025/05/26 by Liang Cheng, Cheng, Liang, Tianyi Li +5
Computer Science · Decision Sciences · #Advanced Data Storage Technologies #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2505.20097

openalex publication_date 2025/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The performance of pre-trained Large Language Models (LLMs) is often sensitive to nuances in prompt templates, requiring careful prompt engineering, adding costs in terms of computing and human effort. In this study, we present experiments encompassing multiple LLMs variants of varying sizes aimed at probing their preference with different prompts. Through experiments on Question Answering, we show prompt preference consistency across LLMs of different sizes. We also show that this consistency extends to other tasks, such as Natural Language Inference. Utilizing this consistency, we propose a method to use a smaller model to select effective prompt templates for a larger model. We show that our method substantially reduces the cost of prompt engineering while consistently matching performance with optimal prompts among candidates. More importantly, our experiment shows the efficacy of our strategy across fourteen LLMs and its applicability to a broad range of NLP tasks, highlighting its robustness

Citations

• Online Prompt Selection for Program Synthesis
• AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimation
• Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
• State of What Art? A Call for Multi-Prompt LLM Evaluation
• Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
• Large Language Models as Optimizers
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Large Language Models
• Learning to Prompt in the Classroom to Understand AI Limits: A pilot study
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Sources of Hallucination by Large Language Models on Inference Tasks
• Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
• Prompting Large Language Model for Machine Translation: A Case Study
• LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Large Language Models Are Human-Level Prompt Engineers
• Prompting GPT-3 To Be Reliable
• Ask Me Anything: A simple strategy for prompting language models
• Entailment Graph Learning with Textual Entailment and Soft Transitivity
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training language models to follow instructions with human feedback
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• Training Verifiers to Solve Math Word Problems
• A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
• Do Prompt-Based Models Really Understand the Meaning of their Prompts?
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
• Language Models for Lexical Inference in Context
• Language Models are Few-Shot Learners
• Negated and Misprimed Probes for Pretrained Language Models: Birds Can\n Talk, But Cannot Fly
• Language Models as Knowledge Bases?
• Probabilistic Models of Relational Implication

Related

• A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks
• Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
• Why is prompting hard? Understanding prompts on binary sequence predictors
• CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulness
• Prompt Exploration with Prompt Regression