Connecting the Dots Between MLE and RL for Sequence Prediction

2018/11/24 by Bowen Tan, Zhiting Hu, Tan, Bowen +7
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1811.09740

openalex publication_date 2018/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently distinct algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of a reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, inspired from the framework, we present a new algorithm that dynamically interpolates among the family of algorithms for scheduled sequence model learning. Experiments on machine translation, text summarization, and game imitation learning demonstrate the superiority of the proposed algorithm.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Reinforcement Learning with Deep Energy-Based Policies
• Equivalence Between Policy Gradients and Soft Q-Learning
• Softmax Q-Distribution Estimation for Structured Prediction: A Theoretical Interpretation for RAML
• Cold-Start Reinforcement Learning with Softmax Policy Gradient
• MaskGAN: Better Text Generation via Filling in the______
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Policy Shaping and Generalized Update Equations for Semantic Parsing\n from Denotations

Cited by

• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

Related

• Reinforcement Learning with Simple Sequence Priors
• From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
• Offline Reinforcement Learning as One Big Sequence Modeling Problem
• ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation
• Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning