More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering

2021/09/25 by Yang Bai, Bai, Yang, Daisy Zhe Wang +1
Computer Science · #68T50 #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2109.12264

openalex publication_date 2021/09/25 · openalex created_date 2021/10/11 · openalex updated_date 2026/07/28

Abstract

Textual Question Answering (QA) aims to provide precise answers to user's questions in natural language using unstructured data. One of the most popular approaches to this goal is machine reading comprehension(MRC). In recent years, many novel datasets and evaluation metrics based on classical MRC tasks have been proposed for broader textual QA tasks. In this paper, we survey 47 recent textual QA benchmark datasets and propose a new taxonomy from an application point of view. In addition, We summarize 8 evaluation metrics of textual QA tasks. Finally, we discuss current trends in constructing textual QA benchmarks and suggest directions for future work.

Citations

• Teaching Machines to Read and Comprehend
• REALM: Retrieval-Augmented Language Model Pre-Training
• Language Models are Few-Shot Learners
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering

Related

• Modern Question Answering Datasets and Benchmarks: A Survey
• FairytaleQA Translated: Enabling Educational Question and Answer Generation in Less-Resourced Languages
• Reading Comprehension as Natural Language Inference: A Semantic Analysis
• Evaluation of Question Answering Systems: Complexity of judging a natural language
• LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA