2022/10/09 by Zhengbao Jiang, Jiang, Zhengbao, Jun Araki +5 · 4 citations
Computer Science · Mathematics · #Arithmetic #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Concatenation (mathematics) #FOS: Computer and information sciences #Generative grammar #Generative model #Hop (telecommunications) #Linguistics #Machine learning #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Question answering #RDF #SPARQL #Topic Modeling #Zero (linguistics) #cs.CL
paper · pdf · doi:10.48550/arxiv.2210.04234
published in arXiv (Cornell University) (Cornell University) · COLING 2022
arxiv created 2022/10/09 · openalex publication_date 2022/10/09 · arxiv updated 2022/10/11 · openalex created_date 2022/10/12 · openalex updated_date 2026/08/08
Generative question answering (QA) models generate answers to questions either solely based on the parameters of the model (the closed-book setting) or additionally retrieving relevant evidence (the open-book setting). Generative QA models can answer some relatively complex questions, but the mechanism through which they do so is still poorly understood. We perform several studies aimed at better understanding the multi-hop reasoning capabilities of generative QA models. First, we decompose multi-hop questions into multiple corresponding single-hop questions, and find marked inconsistency in QA models' answers on these pairs of ostensibly identical question chains. Second, we find that models lack zero-shot multi-hop reasoning ability: when trained only on single-hop questions, models generalize poorly to multi-hop questions. Finally, we demonstrate that it is possible to improve models' zero-shot multi-hop reasoning capacity through two methods that approximate real multi-hop natural language (NL) questions by training on either concatenation of single-hop questions or logical forms (SPARQL). In sum, these results demonstrate that multi-hop reasoning does not emerge naturally in generative QA models, but can be encouraged by advances in training or modeling techniques.