A Deep Reinforcement Learning Chatbot

2017/09/07 by Iulian Vlad Serban, Iulian V. Serban, Serban, Iulian V. +37 · 1 voice · 200 citations
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Chatbot #Computer science #Deep learning #Ensemble learning #Machine learning #Mobile Crowdsensing and Crowdsourcing #Natural language processing #Reinforcement learning #Sequence (biology) #Speech and dialogue systems #Topic Modeling #cs.AI #cs.CL #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1709.02349

published in arXiv (Cornell University) (Cornell University) · 40 pages, 9 figures, 11 tables

openalex publication_date 2017/09/07 · arxiv created 2017/11/05 · arxiv updated 2017/11/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present MILABOT: a deep reinforcement learning chatbot developed by the Montreal Institute for Learning Algorithms (MILA) for the Amazon Alexa Prize competition. MILABOT is capable of conversing with humans on popular small talk topics through both speech and text. The system consists of an ensemble of natural language generation and retrieval models, including template-based models, bag-of-words models, sequence-to-sequence neural network and latent variable neural network models. By applying reinforcement learning to crowdsourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble. The system has been evaluated through A/B testing with real-world users, where it performed significantly better than many competing systems. Due to its machine learning architecture, the system is likely to improve with additional data.

Citations

• Auto-Encoding Variational Bayes
• Matrix Factorization Techniques for Recommender Systems
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• DialPort: Connecting the Spoken Dialog Research Community to Real User Data
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models

Cited by

• Echoing: Identity Failures when LLM Agents Talk to Each Other
• Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
• The Eigenoption-Critic Framework
• Imitation Learning from Imperfect Demonstration
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• An Ensemble Dialogue System for Facts-Based Sentence Generation
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
• Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation
• Towards Explainable and Controllable Open Domain Dialogue Generation with Dialogue Acts
• VisualHints: A Visual-Lingual Environment for Multimodal Reinforcement Learning
• The RLLChatbot: a solution to the ConvAI challenge
• Towards a Human-like Open-Domain Chatbot
• Data-Efficient Methods for Dialogue Systems
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention
• A Deep Reinforcement Learning Chatbot (Short Version)
• Recent advances in conversational NLP : Towards the standardization of Chatbot building
• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
• Just ASK: Building an Architecture for Extensible Self-Service Spoken Language Understanding
• Emotionally-Aware Chatbots: A Survey
• Deploying Lifelong Open-Domain Dialogue Learning
• Learning Representations and Agents for Information Retrieval
• When is ACL's Deadline? A Scientific Conversational Agent
• Science Driven Innovations Powering Mobile Product: Cloud AI vs. Device AI Solutions on Smart Device
• Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning
• Text Generation by Learning from Demonstrations
• Arming the public with artificial intelligence to counter social bots
• Multi-Domain Dialogue Acts and Response Co-Generation
• Synthetic media [wikipedia]

Discussions

• Milabot: A Deep Reinforcement Learning Chatbot [hn, 91 points, 16 comments]

Related

• A Deep Reinforcement Learning Chatbot (Short Version)
• Multi-Purpose NLP Chatbot : Design, Methodology & Conclusion
• The RLLChatbot: a solution to the ConvAI challenge
• An Ensemble Model with Ranking for Social Dialogue
• Deep Learning Based Chatbot Models