Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

2022/12/28 by Omar Khattab, Keshav Santhanam, Khattab, Omar +11 · 36 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2212.14024

openalex publication_date 2022/12/28 · openalex created_date 2023/01/06 · openalex updated_date 2026/07/28

Abstract

Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple "retrieve-then-read" pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37-120%, 8-39%, and 80-290% relative gains against the vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively. We release DSP at https://github.com/stanfordnlp/dsp

Cited by

• A Probabilistic Framework for LLM-Based Model Discovery
• MindWatcher: Toward Smarter Multimodal Tool-Integrated Reasoning
• Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization
• Visual prompt engineering for video models
• FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
• amc: The Automated Mission Classifier for Telescope Bibliographies
• Encoder-Free Knowledge-Graph Reasoning with LLMs via Hyperdimensional Path Retrieval
• LangMark: A Multilingual Dataset for Automatic Post-Editing
• RAG-Driven Data Quality Governance for Enterprise ERP Systems
• Retrieval as a Decision: Training-Free Adaptive Gating for Efficient RAG
• RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAG
• Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems
• Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspective
• Serve Programs, Not Prompts
• Pie: A Programmable Serving System for Emerging LLM Applications
• LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature
• Prompt Decorators: A Declarative and Composable Syntax for Reasoning, Formatting, and Control in LLMs
• SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection
• PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering
• AI-to-AI Feedback: Amplified Intelligence — Prior Art and Governance Implications for Multi-Model Advisory Architectures
• HLTCOE at TREC 2024 NeuCLIR Track
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
• A Large-Scale Dataset and Citation Intent Classification in Turkish with LLMs
• AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humans
• What makes prompts a graph: necessary and sufficient conditions for prompt graph engineering
• A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
• SHERPA: A Model-Driven Framework for Large Language Model Execution
• Instruction-Level Weight Shaping: A Framework for Self-Improving AI Agents
• SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimization
• Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data
• Improving End-to-End Training of Retrieval-Augmented Generation Models via Joint Stochastic Approximation
• Don't Think Twice! Over-Reasoning Impairs Confidence Calibration
• CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation
• A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges
• MAO-ARAG: Multi-Agent Orchestration for Adaptive Retrieval-Augmented Generation
• CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Setting

Related

• REALM: Retrieval-Augmented Language Model Pre-Training
• Large Language Models are Strong Zero-Shot Retriever
• Reliable, Adaptable, and Attributable Language Models with Retrieval
• Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems
• Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture