2021/08/28 by Beomsu Kim, Seokjun Seo, Kim, Beomsu +7
Computer Science · #Artificial Intelligence (cs.AI) #Artificial intelligence #Communication #Computation and Language (cs.CL) #Computer science #Conversation #Domain knowledge #FOS: Computer and information sciences #Generative grammar #Generative model #Inference #Latency (audio) #Machine learning #Natural Language Processing Techniques #Natural language processing #Open domain #Question answering #Scale (ratio) #Set (abstract data type) #Speech and dialogue systems #Test set #Topic Modeling #cs.AI #cs.CL
paper · pdf · doi:10.48550/arxiv.2108.12582
published in arXiv (Cornell University) (Cornell University) · EMNLP21-Findings
openalex publication_date 2021/08/28 · arxiv created 2021/08/31 · arxiv updated 2021/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Despite the remarkable performance of large-scale generative models in open-domain conversation, they are known to be less practical for building real-time conversation systems due to high latency. On the other hand, retrieval models could return responses with much lower latency but show inferior performance to the large-scale generative models since the conversation quality is bounded by the pre-defined response set. To take advantage of both approaches, we propose a new training method called G2R (Generative-to-Retrieval distillation) that preserves the efficiency of a retrieval model while leveraging the conversational ability of a large-scale generative model by infusing the knowledge of the generative model into the retrieval model. G2R consists of two distinct techniques of distillation: the data-level G2R augments the dialogue dataset with additional responses generated by the large-scale generative model, and the model-level G2R transfers the response quality score assessed by the generative model to the score of the retrieval model by the knowledge distillation loss. Through extensive experiments including human evaluation, we demonstrate that our retrieval-based conversation system trained with G2R shows a substantially improved performance compared to the baseline retrieval model while showing significantly lower inference latency than the large-scale generative models.