Triggering Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks

2023/06/06 by Kanishka Misra, Misra, Kanishka, Cícero Nogueira dos Santos +3
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2306.04009

openalex publication_date 2023/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite readily memorizing world knowledge about entities, pre-trained language models (LMs) struggle to compose together two or more facts to perform multi-hop reasoning in question-answering tasks. In this work, we propose techniques that improve upon this limitation by relying on random walks over structured knowledge graphs. Specifically, we use soft prompts to guide LMs to chain together their encoded knowledge by learning to map multi-hop questions to random walk paths that lead to the answer. Applying our methods on two T5 LMs shows substantial improvements over standard tuning approaches in answering questions that require 2-hop reasoning.

Related

• KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration
• Efficient Multi-Hop Question Answering over Knowledge Graphs via LLM Planning and Embedding-Guided Search
• Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning
• Omne-R1: Learning to Reason with Memory for Multi-hop Question Answering
• Semi-Structured Chain-of-Thought: Integrating Multiple Sources of Knowledge for Improved Language Model Reasoning