Variational Reward Estimator Bottleneck: Learning Robust Reward\n Estimator for Multi-Domain Task-Oriented Dialog

2020/05/30 by Jeiyoon Park, Chanhee Lee, Park, Jeiyoon +5
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2006.00417

openalex publication_date 2020/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite its notable success in adversarial learning approaches to\nmulti-domain task-oriented dialog system, training the dialog policy via\nadversarial inverse reinforcement learning often fails to balance the\nperformance of the policy generator and reward estimator. During optimization,\nthe reward estimator often overwhelms the policy generator and produces\nexcessively uninformative gradients. We proposes the Variational Reward\nestimator Bottleneck (VRB), which is an effective regularization method that\naims to constrain unproductive information flows between inputs and the reward\nestimator. The VRB focuses on capturing discriminative features, by exploiting\ninformation bottleneck on mutual information. Empirical results on a\nmulti-domain task-oriented dialog dataset demonstrate that the VRB\nsignificantly outperforms previous methods.\n

Citations

• The information bottleneck method
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
• Generative Adversarial Imitation Learning
• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• User Modeling for Task Oriented Dialogues

Related

• Guided Dialog Policy Learning: Reward Estimation for Multi-Domain\n Task-Oriented Dialog
• Adversarial Learning of Task-Oriented Neural Dialog Models
• Why Guided Dialog Policy Learning performs well? Understanding the role of adversarial learning and its alternative
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
• Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems