Reinforcement Learning for Bandit Neural Machine Translation with\n Simulated Human Feedback

2017/07/24 by Khanh Nguyen, Hal Daumé, Nguyen, Khanh +3 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1707.07402

openalex publication_date 2017/07/24 · openalex created_date 2022/10/04 · openalex updated_date 2026/07/28

Abstract

Machine translation is a natural candidate problem for reinforcement learning\nfrom human feedback: users provide quick, dirty ratings on candidate\ntranslations to guide a system to improve. Yet, current neural machine\ntranslation training focuses on expensive human-generated reference\ntranslations. We describe a reinforcement learning algorithm that improves\nneural machine translation systems from simulated human feedback. Our algorithm\ncombines the advantage actor-critic algorithm (Mnih et al., 2016) with the\nattention-based neural encoder-decoder architecture (Luong et al., 2015). This\nalgorithm (a) is well-designed for problems with a large action space and\ndelayed rewards, (b) effectively optimizes traditional corpus-level machine\ntranslation metrics, and (c) is robust to skewed, high-variance, granular\nfeedback modeled after actual human behaviors.\n

Cited by

• Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
• Learning to Reason as Action Abstractions with Scalable Mid-Training RL
• Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security

Related

• A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation
• Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning
• Can Neural Machine Translation be Improved with User Feedback?
• The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task
• Neural Machine Translation with Adequacy-Oriented Learning