Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus

2018/02/07 by Jianxiong Dong, Dong, Jianxiong, Jim Huang +1 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1802.02614

openalex publication_date 2018/02/07 · arxiv created 2018/05/07 · arxiv updated 2018/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Ubuntu dialogue corpus is the largest public available dialogue corpus to make it feasible to build end-to-end deep neural network models directly from the conversation data. One challenge of Ubuntu dialogue corpus is the large number of out-of-vocabulary words. In this paper we proposed a method which combines the general pre-trained word embedding vectors with those generated on the task-specific training set to address this issue. We integrated character embedding into Chen et al's Enhanced LSTM method (ESIM) and used it to evaluate the effectiveness of our proposed method. For the task of next utterance selection, the proposed method has demonstrated a significant performance improvement against original ESIM and the new model has achieved state-of-the-art results on both Ubuntu dialogue corpus and Douban conversation corpus. In addition, we investigated the performance impact of end-of-utterance and end-of-turn token tags.

Citations

• Adam: A Method for Stochastic Optimization
• Character-Aware Neural Language Models
• Long Short-Term Memory
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• Sentence Pair Scoring: Towards Unified Framework for Text Comprehension
• An Ensemble Method to Produce High-Quality Word Embeddings (2016)
• Words or Characters? Fine-grained Gating for Reading Comprehension
• Convolutional Neural Networks for Sentence Classification
• A Decomposable Attention Model for Natural Language Inference
• Enriching Word Vectors with Subword Information

Cited by

• Pre-Trained and Attention-Based Neural Networks for Building Noetic Task-Oriented Dialogue Systems
• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots

Related

• Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
• The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
• Building Sequential Inference Models for End-to-End Response Selection
• Exploiting Sentence and Context Representations in Deep Neural Models for Spoken Language Understanding
• End-to-end Conversation Modeling Track in DSTC6