Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

2023/10/23 by Kim, Gangwoo, Kim, Sungdong, Jeon, Byeongguk +2 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2310.14696

Abstract

Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge. To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC): It recursively constructs a tree of disambiguations for the AQ -- via few-shot prompting leveraging external knowledge -- and uses it to generate a long-form answer. ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.

Cited by

• Leveraging Spreading Activation for Improved Document Retrieval in Knowledge-Graph-Based RAG Systems
• Learning Steerable Clarification Policies with Collaborative Self-play

Related

• Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions
• Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
• Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation
• AmbigQA: Answering Ambiguous Open-domain Questions
• CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models