UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation

2023/03/15 by Daixuan Cheng, Cheng, Daixuan, Shaohan Huang +17 · 16 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2303.08518

openalex publication_date 2023/03/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) are popular for their impressive abilities, but the need for model-specific fine-tuning or task-specific prompt engineering can hinder their generalization. We propose UPRISE (Universal Prompt Retrieval for Improving zero-Shot Evaluation), which tunes a lightweight and versatile retriever that automatically retrieves prompts for a given zero-shot task input. Specifically, we demonstrate universality in a cross-task and cross-model scenario: the retriever is tuned on a diverse set of tasks, but tested on unseen task types; we use a small frozen LLM, GPT-Neo-2.7B, for tuning the retriever, but test the retriever on different LLMs of much larger scales, such as BLOOM-7.1B, OPT-66B and GPT3-175B. Additionally, we show that UPRISE mitigates the hallucination problem in our experiments with ChatGPT, suggesting its potential to improve even the strongest LLMs. Our model and code are available at https://github.com/microsoft/LMOps.

Cited by

• Building Specialized Software-Assistant ChatBot with Graph-Based Retrieval-Augmented Generation
• Data-Model Co-Evolution: Growing Test Sets to Refine LLM Behavior
• Large Language Models Hallucination: A Comprehensive Survey
• Topic Coverage-based Demonstration Retrieval for In-Context Learning
• InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• TDR: Task-Decoupled Retrieval with Fine-Grained LLM Feedback for In-Context Learning
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
• A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models
• MMSearch-R1: Incentivizing LMMs to Search
• Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds
• Learning to Select In-Context Demonstration Preferred by Large Language Model
• Spoken Language Understanding on Unseen Tasks With In-Context Learning
• AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Models
• TelcoAI: Advancing 3GPP Technical Specification Search through Agentic Multi-Modal Retrieval-Augmented Generation
• Fashion-RAG: Multimodal Fashion Image Editing via Retrieval-Augmented Generation

Related

• Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance
• Large Language Models are Strong Zero-Shot Retriever
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning
• Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models