Deep Reinforcement Learning for Dialogue Generation

2016/06/05 by Jiwei Li, Li, Jiwei, Will Monroe +9 · 61 citations
Computer Science · #AI in Service Interactions #Computation and Language (cs.CL) #FOS: Computer and information sciences #Speech and dialogue systems #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1606.01541

openalex publication_date 2016/06/05 · arxiv created 2016/09/29 · arxiv updated 2016/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.

Citations

• A Neural Conversational Model
• Playing Atari with Deep Reinforcement Learning
• Attention with Intention for a Neural Network Conversation Model
• Neural Responding Machine for Short-Text Conversation
• Sequence Level Training with Recurrent Neural Networks
• Reinforcement Learning Neural Turing Machines - Revised
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Persona-Based Neural Conversation Model
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Continuously Learning Neural Dialogue Management
• Language Understanding for Text-based Games Using Deep Reinforcement\n Learning
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

Cited by

• Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
• Online Robust Reinforcement Learning with General Function Approximation
• SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• Towards better dense rewards in Reinforcement Learning Applications
• Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO
• Knowing Ourselves Through Others: Reflecting with AI in Digital Human Debates
• ECO Decoding: Entropy-Based Control for Controllability and Fluency in Controllable Dialogue Generation
• Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks
• TensorBLEU: Vectorized GPU-based BLEU Score Implementation for Per-Sentence In-Training Evaluation
• MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generation
• HumAIne-Chatbot: Real-Time Personalized Conversational AI via Reinforcement Learning
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
• Collaborative Rational Speech Act: Pragmatic Reasoning for Multi-Turn Dialog
• SoK: Machine Unlearning for Large Language Models
• Optimizing Conversational Product Recommendation via Reinforcement Learning
• Tailored Conversations beyond LLMs: A RL-Based Dialogue Manager
• Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation
• History-Aware Cross-Attention Reinforcement: Self-Supervised Multi Turn and Chain-of-Thought Fine-Tuning with vLLM
• GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Dialog State Tracking with Reinforced Data Augmentation
• Zero-Resource Knowledge-Grounded Dialogue Generation
• Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy
• ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models
• Automatic, Personalized, and Flexible Playlist Generation using Reinforcement Learning
• Learning Natural Language Generation from Scratch
• Def-DTS: Deductive Reasoning for Open-domain Dialogue Topic Segmentation
• Distantly Supervised Question Parsing
• Distributed Representation for Traditional Chinese Medicine Herb via Deep Learning Models
• SCAR: Shapley Credit Assignment for More Efficient RLHF
• Latent Intention Dialogue Models
• DialogAct2Vec: Towards End-to-End Dialogue Agent by Multi-Task Representation Learning
• Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text
• Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
• RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving
• Generalized Coarse-to-Fine Visual Recognition with Progressive Training
• Ekova: A Personality-Support Agent for Self-Discovery Dialogue
• Shaping the Narrative Arc: An Information-Theoretic Approach to\n Collaborative Dialogue
• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts
• End-to-End Task-Completion Neural Dialogue Systems
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Adversarial Bootstrapping for Dialogue Model Training
• Science Driven Innovations Powering Mobile Product: Cloud AI vs. Device AI Solutions on Smart Device
• Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
• Controllable Dialogue Generation with Disentangled Multi-grained Style Specification and Attribute Consistency Reward
• End-to-End Trainable Non-Collaborative Dialog System
• Know More about Each Other: Evolving Dialogue Strategy via Compound Assessment
• Generative Deep Neural Networks for Dialogue: A Short Review
• NGENT: Next-Generation AI Agents Must Integrate Multi-Domain Abilities to Achieve Artificial General Intelligence
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Reinforcement Learning Based Emotional Editing Constraint Conversation Generation
• Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems
• Graph-Stega: Semantic Controllable Steganographic Text Generation Guided by Knowledge Graph
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots
• Chat More If You Like: Dynamic Cue Words Planning to Flow Longer Conversations

Related

• Deep Reinforcement Learning For Modeling Chit-Chat Dialog With Discrete Attributes
• Say What I Want: Towards the Dark Side of Neural Dialogue Models
• Cascaded LSTMs based Deep Reinforcement Learning for Goal-driven Dialogue
• A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots
• Deep Reinforcement Learning for Conversational AI