Reward Augmented Maximum Likelihood for Neural Structured Prediction

2016/09/01 by Mohammad Norouzi, Norouzi, Mohammad, Samy Bengio +11 · 9 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1609.00150

openalex publication_date 2016/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation. This paper presents a simple and computationally efficient approach to incorporate task reward into a maximum likelihood framework. By establishing a link between the log-likelihood and expected reward objectives, we show that an optimal regularized expected reward is achieved when the conditional distribution of the outputs given the inputs is proportional to their exponentiated scaled rewards. Accordingly, we present a framework to smooth the predictive probability of the outputs using their corresponding rewards. We optimize the conditional log-probability of augmented outputs that are sampled proportionally to their exponentiated scaled rewards. Experiments on neural sequence to sequence models for speech recognition and machine translation show notable improvements over a maximum likelihood baseline by using reward augmented maximum likelihood (RAML), where the rewards are defined as the negative edit distance between the outputs and the ground truth labels.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Kaldi Speech Recognition Toolkit
• Loss-sensitive Training of Probabilistic Conditional Random Fields
• Gradient-based learning applied to document recognition
• Sequence to Sequence Learning with Neural Networks
• Sequence-to-Sequence Learning as Beam-Search Optimization
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning

Cited by

• Towards better dense rewards in Reinforcement Learning Applications
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation
• Analysing Dropout and Compounding Errors in Neural Language Models
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• From Scarcity to Efficiency: Investigating the Effects of Data Augmentation on African Machine Translation
• Structuring GUI Elements through Vision Language Models: Towards Action Space Generation
• Incremental Learning with Maximum Entropy Regularization: Rethinking\n Forgetting and Intransigence
• A Reward-Directed Diffusion Framework for Generative Design Optimization

Related

• Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• Bandit Structured Prediction for Neural Sequence-to-Sequence Learning
• Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
• From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation