2018/09/25 by Chengwei Su, Rahul Gupta, Su, Chengwei +5 · 1 citation
Computer Science · Mathematics · #Artificial intelligence #Calibration #Classifier (UML) #Computation and Language (cs.CL) #Computer science #Database #Domain (mathematical analysis) #FOS: Computer and information sciences #Mathematics #Natural Language Processing Techniques #Natural language #Natural language processing #Natural language understanding #Scalability #Speech and dialogue systems #Statistics #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.1809.09605
7 pages, Accepted to IEEE SLT-2018
arxiv created 2018/09/25 · openalex publication_date 2018/09/25 · arxiv updated 2018/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Large scale Natural Language Understanding (NLU) systems are typically trained on large quantities of data, requiring a fast and scalable training strategy. A typical design for NLU systems consists of domain-level NLU modules (domain classifier, intent classifier and named entity recognizer). Hypotheses (NLU interpretations consisting of various intent+slot combinations) from these domain specific modules are typically aggregated with another downstream component. The re-ranker integrates outputs from domain-level recognizers, returning a scored list of cross domain hypotheses. An ideal re-ranker will exhibit the following two properties: (a) it should prefer the most relevant hypothesis for the given input as the top hypothesis and, (b) the interpretation scores corresponding to each hypothesis produced by the re-ranker should be calibrated. Calibration allows the final NLU interpretation score to be comparable across domains. We propose a novel re-ranker strategy that addresses these aspects, while also maintaining domain specific modularity. We design optimization loss functions for such a modularized re-ranker and present results on decreasing the top hypothesis error rate as well as maintaining the model calibration. We also experiment with an extension involving training the domain specific re-rankers on datasets curated independently by each domain to allow further asynchronization. %The proposed re-ranker design showcases the following: (i) improved NLU performance over an unweighted aggregation strategy, (ii) cross-domain calibrated performance and, (iii) support for use cases involving training each re-ranker on datasets curated by each domain independently.