2021/01/06 by Sofian Chaybouti, Chaybouti, Sofian, Achraf Saghe +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2101.02157
openalex publication_date 2021/01/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
State-of-the-art extractive question-answering models achieve superhuman performances on the SQuAD benchmark. Yet, they are unreasonably heavy and need expensive GPU computing to answer questions in a reasonable time. Thus, they cannot be used in the open-domain question-answering paradigm for real-world queries on hundreds of thousands of documents. In this paper, we explore the possibility of transferring the natural language understanding of language models into dense vectors representing questions and answer candidates to make question-answering compatible with a simple nearest neighbor search task. This new model, which we call EfficientQA, takes advantage of the pair of sequences kind of input of BERT-based models to build meaningful, dense representations of candidate answers. These latter are extracted from the context in a question-agnostic fashion. Our model achieves state-of-the-art results in Phrase-Indexed Question Answering (PIQA), beating the previous state-of-art by 1.3 points in exact-match and 1.4 points in f1-score. These results show that dense vectors can embed rich semantic representations of sequences, although these were built from language models not originally trained for the use case. Thus, to build more resource-efficient NLP systems in the future, training language models better adapted to build dense representations of phrases is one of the possibilities.