Self-Knowledge Guided Retrieval Augmentation for Large Language Models

2023/10/08 by Yile Wang, Wang, Yile, Peng Li +5 · 8 citations
Computer Science · #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.05002

openalex publication_date 2023/10/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called self-knowledge) and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either InstructGPT or ChatGPT.

Cited by

• Leveraging Spreading Activation for Improved Document Retrieval in Knowledge-Graph-Based RAG Systems
• A Systematic Framework for Enterprise Knowledge Retrieval: Leveraging LLM-Generated Metadata to Enhance RAG Systems
• Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection
• Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs
• KnowRL: Teaching Language Models to Know What They Know
• Can an LLM Induce a Graph? Investigating Memory Drift and Context Length
• Impact-driven Context Filtering For Cross-file Code Completion
• MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation

Related

• SKILL-RAG: Self-Knowledge Induced Learning and Filtering for Retrieval-Augmented Generation
• CCSK:Cognitive Convection of Self-Knowledge Based Retrieval Augmentation for Large Language Models
• RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback
• Do Large Language Models Know What They Don't Know?
• Self-Routing RAG: Binding Selective Retrieval with Knowledge Verbalization