RAC: Efficient LLM Factuality Correction with Retrieval Augmentation

2024/10/21 by Changmao Li, Li, Changmao, Jeffrey Flanigan +1 · 2 citations
Computer Science · Decision Sciences · #Natural Language Processing Techniques #Topic Modeling #Data Quality and Management

paper · pdf · doi:10.48550/arxiv.2410.15667

Abstract

Large Language Models (LLMs) exhibit impressive results across a wide range of natural language processing (NLP) tasks, yet they can often produce factually incorrect outputs. This paper introduces a simple but effective low-latency post-correction method, Retrieval Augmented Correction (RAC), aimed at enhancing the factual performance of LLMs without requiring additional fine-tuning. Our method is general and can be used with any instruction-tuned LLM, and has greatly reduced latency compared to prior approaches. RAC decomposes the LLM's output into atomic facts and applies a fine-grained verification and correction process with retrieved content to verify and correct the LLM-generated output. Our extensive experiments show that RAC yields up to 30% improvements over state-of-the-art baselines across two popular factuality evaluation datasets, validating its efficacy and robustness in both with and without the integration of Retrieval-Augmented Generation (RAG) across different LLMs.\footnoteOur code is at \urlhttps://github.com/jlab-nlp/Retrieval-Augmented-Correction

Cited by

• SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA
• Mathematical Analysis of Hallucination Dynamics in Large Language Models: Uncertainty Quantification, Advanced Decoding, and Principled Mitigation

Related

• UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models
• FaaF: Facts as a Function for the evaluation of generated text
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
• Fine-tuning Language Models for Factuality
• Post-training an LLM for RAG? Train on Self-Generated Demonstrations