Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge

2024/12/11 by Zhang, Haotong
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2412.08317

openalex publication_date 2024/12/11 · openalex created_date 2024/12/13 · openalex updated_date 2026/07/28

Abstract

We carry out a series of experiments to test large language models' multi-hop reasoning ability from three aspects: selecting and combining external knowledge, dealing with non-sequential reasoning tasks and generalising to data samples with larger numbers of hops. We test the GPT-3.5 model on four reasoning benchmarks with Chain-of-Thought prompting (and its variations). Our results reveal that despite the amazing performance achieved by large language models on various reasoning tasks, models still suffer from severe drawbacks which shows a large gap with humans.

Related

• Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?
• Do Large Language Models Latently Perform Multi-Hop Reasoning?
• Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
• Specializing Smaller Language Models towards Multi-Step Reasoning
• Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models