LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering

2024/10/23 by Qi Zhao, Zhao, Qingfei, Ruobing Wang +10 · 10 citations
Computer Science · #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2410.18050

openalex publication_date 2024/10/23 · openalex created_date 2024/11/14 · openalex updated_date 2026/07/28

Abstract

Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. Data and code are available at https://github.com/QingFei1/LongRAG.

Cited by

• Metadata-Driven Retrieval-Augmented Generation for Financial Question Answering
• A Survey of Data Agents: Emerging Paradigm or Overstated Hype?
• VoiceAgentBench: Are Voice Assistants ready for agentic tasks?
• VersionRAG: Version-Aware Retrieval-Augmented Generation for Evolving Documents
• Evaluating the Sensitivity of LLMs to Harmful Contents in Long Input
• AuditAgent: Expert-Guided Multi-Agent Reasoning for Cross-Document Fraudulent Evidence Discovery
• Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaboration
• Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference
• Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG
• Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning

Related

• Long Context vs. RAG for LLMs: An Evaluation and Revisits
• In Defense of RAG in the Era of Long-Context Language Models
• ALR²: A Retrieve-then-Reason Framework for Long-context Question Answering
• Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
• Long²RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall