MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

2024/01/27 by Yixuan Tang, Yi Yang, Tang, Yixuan +1 · 47 citations
Computer Science · #Advanced Data Compression Techniques #Algorithms and Data Compression #Caching and Content Delivery #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2401.15391

openalex publication_date 2024/01/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. The MultiHop-RAG and implemented RAG system is publicly available at https://github.com/yixuantt/MultiHop-RAG/.

Cited by

• LIR³AG: A Lightweight Rerank Reasoning Strategy Framework for Retrieval-Augmented Generation
• Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias
• An Index-based Approach for Efficient and Effective Web Content Extraction
• Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems
• NeuroPath: Neurobiology-Inspired Path Tracking and Reflection for Semantically Coherent Retrieval
• Structured RAG for Answering Aggregative Questions
• ContextPilot: Fast Long-Context Inference via Context Reuse
• Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval
• MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval
• BambooKG: A Neurobiologically-inspired Frequency-Weight Knowledge Graph
• OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora
• SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models
• Verification-Aware Planning for Multi-Agent Systems
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• PluriHop: Exhaustive, Recall-Sensitive QA over Distractor-Rich Corpora
• PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering
• RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge
• CTIArena: Benchmarking LLM Knowledge and Reasoning Across Heterogeneous Cyber Threat Intelligence
• Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
• DeepResearchGuard: Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safety
• Cluster-based Adaptive Retrieval: Dynamic Context Selection for RAG Applications
• Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective
• When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs
• Reasoning by Exploration: A Unified Approach to Retrieval and Generation over Graphs
• KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension
• FinReflectKG -- MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence
• ReTAG: Retrieval-Enhanced, Topic-Augmented Graph-Based Global Sensemaking
• Beyond Static Retrieval: Opportunities and Pitfalls of Iterative Retrieval in GraphRAG
• MIRAGE: Multi-hop Reasoning with Ambiguity Evaluation for Illusory Questions
• Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
• Query-Centric Graph Retrieval Augmented Generation
• GLM-RAG: Graph Language Models for Graph-Based Retrieval-Augmented Generation
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution
• GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation
• Aligning LLMs for the Classroom with Knowledge-Based Retrieval -- A Comparative RAG Study
• Compare: A Framework for Scientific Comparisons
• OPERA: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval
• JudgeAgent: Knowledge-wise and Dynamic LLM Evaluation with Agent-as-Interviewer
• MSRS: Evaluating Multi-Source Retrieval-Augmented Generation
• Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information
• A Question Answering Dataset for Temporal-Sensitive Retrieval-Augmented Generation
• Can we Evaluate RAGs with Synthetic Data?
• ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning
• Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
• Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
• Beyond Chunks and Graphs: Retrieval-Augmented Generation through Triplet-Driven Thinking
• From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs

Related

• Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
• BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression
• Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models
• Transforming Questions and Documents for Semantically Aligned Retrieval-Augmented Generation
• Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers