Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
2024/07/23 by Li, Zhuowan, Li, Cheng, Zhang, Mingyang +2 · 53 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2407.16833
Abstract
Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.
Cited by
- Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments
- Adaptation of Embedding Models to Financial Filings via LLM Distillation
- CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning
- RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
- A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
- Evaluation of retrieval-based QA on QUEST-LOFT
- ContextPilot: Fast Long-Context Inference via Context Reuse
- ToM: Leveraging Tree-oriented MapReduce for Long-Context Reasoning in Large Language Models
- SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models
- Metadata-Driven Retrieval-Augmented Generation for Financial Question Answering
- Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attention
- Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs
- Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
- VersionRAG: Version-Aware Retrieval-Augmented Generation for Evolving Documents
- vAttention: Verified Sparse Attention
- LongCodeZip: Compress Long Context for Code Language Models
- Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
- GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation
- Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning
- Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering
- Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
- Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generations
- A Theory of Adaptive Scaffolding for LLM-Based Pedagogical Agents
- A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
- Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection
- Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k
- Knowledge Fusion via Bidirectional Information Aggregation
- LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework
- Leveraging Reference Documents for Zero-Shot Ranking via Large Language Models
- Maximally-Informative Retrieval for State Space Model Generation
- Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges
- MIRIAD: Augmenting LLMs with millions of medical query-response pairs
- Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
- Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
- Rethinking Hybrid Retrieval: When Small Embeddings and LLM Re-ranking Beat Bigger Models
- QwenLong-CPRS: Towards ∞-LLMs with Dynamic Context Optimization
- FB-RAG: Improving RAG with Forward and Backward Lookup
- ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts
- Know3-RAG: A Knowledge-aware RAG Framework with Adaptive Retrieval, Generation, and Filtering
- Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation
- TARGET: Benchmarking Table Retrieval for Generative Tasks
- Putting It All into Context: Simplifying Agents with LCLMs
- Overflow Prevention Enhances Long-Context Recurrent LLMs
- Uncertainty-Aware Hybrid Retrieval for Long-Document RAG
- An Agentic Approach to Metadata Reasoning
- Coding Agents are Effective Long-Context Processors
- FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation
- Reflexive Prompt Engineering: A Framework for Responsible Prompt Engineering and Interaction Design
- Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarization
- Single-Pass Document Scanning for Question Answering
- A Survey of Scaling in Large Language Model Reasoning
Related