Large Language Models Are Human-Level Prompt Engineers

2022/11/03 by Yongchao Zhou, Andrei Ioan Muresanu, Zhou, Yongchao +11 · 3 voices · 126 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering Research #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2211.01910

openalex publication_date 2022/11/03 · arxiv published 2022/11/03 · arxiv updated 2023/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Experiments on 24 NLP tasks show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 19/24 tasks. We conduct extensive qualitative and quantitative analyses to explore the performance of APE. We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts. Please check out our webpage at https://sites.google.com/view/automatic-prompt-engineer.

Cited by

• A Toolbox for Improving Evolutionary Prompt Search
• Discrete prompt optimization using genetic algorithm for secure Python code generation
• Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
• Verbalized Particle Posterior: Bayesian Inference over Natural Language Hypotheses
• EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff
• In-Context Learning as Implicit Policy Gradient
• Keyword Matters: Unveiling the Energy Sensitivity of On-Device LLM Prompting
• MAction-SocialNav: Multi-Action Socially Compliant Navigation via Reasoning-enhanced Prompt Tuning
• GateBreaker: Gate-Guided Attacks on Mixture-of-Expert LLMs
• FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
• Unified Multimodal Brain Decoding via Cross-Subject Soft-ROI Fusion
• Auto-Prompting with Retrieval Guidance for Frame Detection in Logistics
• CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs
• LLMs on Drugs: Language Models Are Few-Shot Consumers
• Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queries
• Sharing State Between Prompts and Programs
• Verification-Guided Context Optimization for Tool Calling via Hierarchical LLMs-as-Editors
• Authors Should Label Their Own Documents
• Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptation
• Agile Deliberation: Concept Deliberation for Subjective Visual Classification
• PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data
• Evolving Excellence: Automated Optimization of LLM-based Agents
• HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems
• Empirical Prompt Engineering for Construct Identification with Large Language Models
• promptolution: A Unified, Modular Framework for Prompt Optimization
• Agentic Policy Optimization via Instruction-Policy Co-Evolution
• E-GEO: A Testbed for Generative Engine Optimization in E-Commerce
• Structured Prompting Enables More Robust Evaluation of Language Models
• LLM-Driven Transient Stability Assessment: From Automated Simulation to Neural Architecture Design
• Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Level
• Prompt Optimization as a State-Space Search Problem
• MultiGA: Leveraging Multi-Source Seeding in Genetic Algorithms
• ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
• Reflexive Evidence-Based Multimodal Learning for Clean Energy Transitions: Causal Insights on Cooking Fuel Access, Urbanization, and Carbon Emissions
• GPS: General Per-Sample Prompter
• Dynamic Template Selection for Output Token Generation Optimization: MLP-Based and Transformer Approaches
• RAG-Driven Data Quality Governance for Enterprise ERP Systems
• Lacking Data? No worries! How synthetic images can alleviate image scarcity in wildlife surveys: a case study with muskox (Ovibos moschatus)
• Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models
• Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks
• Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat
• QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code
• SAGE: Stochastic Prompt Optimization via Agent-Guided Exploration
• optimize_anything: Unified Text Optimization can Outperform Specialized Systems
• PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs
• Auto prompting without training labels: An LLM cascade for product quality assessment in e-commerce catalogs
• Compiler.next: A Search-Based Compiler to Power the AI-Native Future of Software Engineering
• Alita-G: Self-Evolving Generative Agent for Agent Generation
• Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI
• How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation
• Code-enabled language models can outperform reasoning models on diverse tasks
• Enhancing Hotel Recommendations with AI: LLM-Based Review Summarization and Query-Driven Insights
• The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives
• Video Reasoning without Training
• DRO-InstructZero: Distributionally Robust Prompt Optimization for Large Language Models
• Addressing the alignment problem in transportation policy making: an LLM approach
• EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems
• LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization
• PromptFlow: Training Prompts Like Neural Networks
• Promptimizer: User-Led Prompt Optimization for Personal Content Classification
• Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs
• On the Relationship Between the Choice of Representation and In-Context Learning
• Bug Histories as Sources of Compiler Fuzzing Mutators
• Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priors
• Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Tasks
• ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation
• Learning to Rewrite Prompts for Bootstrapping LLMs on Downstream Tasks
• Auto-Prompt Ensemble for LLM Judge
• Prompt reinforcing for long-term planning of large language models
• ARM: Discovering Agentic Reasoning Modules for Generalizable Multi-Agent Systems
• Prompting in Practice: Investigating Software Practitioners' Use of Generative AI Tools
• RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems
• AI-to-AI Feedback: Amplified Intelligence — Prior Art and Governance Implications for Multi-Model Advisory Architectures
• LLM Based Bayesian Optimization for Prompt Search
• Structuring Reasoning for Complex Rules Beyond Flat Representations
• ACT: Agentic Classification Tree
• Using GPT to build a Project Management assistant for Jira environments
• FedPOB: Sample-Efficient Federated Prompt Optimization via Bandits
• Your thoughts tell who you are: Characterize the reasoning patterns of LRMs
• The Impact of Role Design in In-Context Learning for Large Language Models
• No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimization
• Following the TRACE: A Structured Path to Empathetic Response Generation with Multi-Agent Models
• An LLM-Powered Agent for Real-Time Analysis of the Vietnamese IT Job Market
• ReviewScore: Misinformed Peer Review Detection with Large Language Models
• Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond
• Distilling Many-Shot In-Context Learning into a Cheat Sheet
• Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning
• Cybersecurity Detection Classification with Reasoning-enabled Language Models
• Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems
• AutoMem: Automated Learning of Memory as a Cognitive Skill
• UniAPO: Unified Multimodal Automated Prompt Optimization
• LAD-VF: LLM-Automatic Differentiation Enables Fine-Tuning-Free Robot Planning from Formal Methods Feedback
• Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
• ZERA: Zero-init Instruction Evolving Refinement Agent -- From Zero Instructions to Structured Prompts via Principle-based Optimization
• PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimization
• NeuroStrike: Neuron-Level Attacks on Aligned LLMs
• Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration
• Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulations
• Uncovering Scaling Laws for Large Language Models via Inverse Problems
• Characterizing Fitness Landscape Structures in Prompt Engineering
• PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs
• Better by Comparison: Retrieval-Augmented Contrastive Reasoning for Automatic Prompt Optimization
• AMAZe: A Multi-Agent Zero-shot Index Advisor for Relational Databases
• Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Prompting
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework
• MQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
• Membership Inference Attacks on LLM-based Recommender Systems
• Automatic Prompt Optimization with Prompt Distillation
• Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data
• Prompt Orchestration Markup Language
• Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation
• CryptoScope: Utilizing Large Language Models for Automated Cryptographic Logic Vulnerability Detection
• Reasoning Steps as Curriculum: Using Depth of Thought as a Difficulty Signal for Tuning LLMs
• APIO: Automatic Prompt Induction and Optimization for Grammatical Error Correction and Text Simplification
• GreenTEA: Gradient Descent with Topic-modeling and Evolutionary Auto-prompting
• When the Domain Expert Has No Time and the LLM Developer Has No Clinical Expertise: Real-World Lessons from LLM Co-Design in a Safety-Net Hospital
• The Problem of Atypicality in LLM-Powered Psychiatry
• Cyberbullying Detection via Aggression-Enhanced Prompting
• Multi-module GRPO: Composing Policy Gradients and Prompt Optimization for Language Model Programs
• LatentPrompt: Optimizing Promts in Latent Space
• MOPrompt: Multi-objective Semantic Evolution for Prompt Optimization
• Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspective
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model
• Automatically discovering heuristics in a complex SAT solver with large language models
• XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML
• From Prompt to Pipeline: Large Language Models for Scientific Workflow Development in Bioinformatics

Discussions

• Large Language Models Are Human-Level Prompt Engineers [hn, 3 points, 0 comments]
• Here is an interesting related paper on *natural language program synthesis* (programs in natural language executed by an LLM). arxiv.org/abs/2211.01910 [bsky, 1 points, 0 comments]
• Yo these robot are already out here trying to take those FRESH “prompt engineer” jobs 🤖 There goes my future 300k 🥲 https://arxiv.org/pdf/2211.01910.pdf [bsky, 0 points, 0 comments]

Related

• Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models
• A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks
• Autonomous Prompt Engineering in Large Language Models
• Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey
• Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code