A short variational proof of equivalence between policy gradients and soft Q learning

2017/12/22 by Pierre H. Richemond, Richemond, Pierre H., Brendan Maginnis +1 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.LG

paper · pdf · doi:10.48550/arxiv.1712.08650

arxiv created 2017/12/22 · arxiv updated 2017/12/27

Abstract

Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this result to the well-known convex duality of Shannon entropy and the softmax function. Such a result is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning.

Cited by

• Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

Related

• Equivalence Between Policy Gradients and Soft Q-Learning
• Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning
• Elementary Analysis of Policy Gradient Methods
• Policy Optimization Reinforcement Learning with Entropy Regularization
• Bridging the Gap Between Value and Policy Based Reinforcement Learning