2020/10/21 by Mohammad Kachuee, Hao Yuan, Kachuee, Mohammad +5
Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human Mobility and Location-Based Analysis #Machine Learning (cs.LG) #Recommender Systems and Techniques #Speech and dialogue systems
paper · pdf · doi:10.48550/arxiv.2010.11230
openalex publication_date 2020/10/21 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Turn-level user satisfaction is one of the most important performance metrics\nfor conversational agents. It can be used to monitor the agent's performance\nand provide insights about defective user experiences. Moreover, a powerful\nsatisfaction model can be used as an objective function that a conversational\nagent continuously optimizes for. While end-to-end deep learning has shown\npromising results, having access to a large number of reliable annotated\nsamples required by these methods remains challenging. In a large-scale\nconversational system, there is a growing number of newly developed skills,\nmaking the traditional data collection, annotation, and modeling process\nimpractical due to the required annotation costs as well as the turnaround\ntimes. In this paper, we suggest a self-supervised contrastive learning\napproach that leverages the pool of unlabeled data to learn user-agent\ninteractions. We show that the pre-trained models using the self-supervised\nobjective are transferable to the user satisfaction prediction. In addition, we\npropose a novel few-shot transfer learning approach that ensures better\ntransferability for very small sample sizes. The suggested few-shot method does\nnot require any inner loop optimization process and is scalable to very large\ndatasets and complex models. Based on our experiments using real-world data\nfrom a large-scale commercial system, the suggested approach is able to\nsignificantly reduce the required number of annotations, while improving the\ngeneralization on unseen out-of-domain skills.\n