2021/08/05 by Haytham ElFadeel, ElFadeel, Haytham, Stan Peshterliev +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2108.02765
openalex publication_date 2021/08/05 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Large transformer models, such as BERT, achieve state-of-the-art results in\nmachine reading comprehension (MRC) for open-domain question answering (QA).\nHowever, transformers have a high computational cost for inference which makes\nthem hard to apply to online QA systems for applications like voice assistants.\nTo reduce computational cost and latency, we propose decoupling the transformer\nMRC model into input-component and cross-component. The decoupling allows for\npart of the representation computation to be performed offline and cached for\nonline use. To retain the decoupled transformer accuracy, we devised a\nknowledge distillation objective from a standard transformer model. Moreover,\nwe introduce learned representation compression layers which help reduce by\nfour times the storage requirement for the cache. In experiments on the SQUAD\n2.0 dataset, a decoupled transformer reduces the computational cost and latency\nof open-domain MRC by 30-40% with only 1.2 points worse F1-score compared to a\nstandard transformer.\n