Revisit Policy Optimization in Matrix Form

2019/09/19 by Luan, Sitao, Chang, Xiao-Wen, Precup, Doina · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1909.09186

Abstract

In tabular case, when the reward and environment dynamics are known, policy evaluation can be written as \bmV_\bmπ = (I - γP_\bmπ)^-1 \bmr_\bmπ, where P_\bmπ is the state transition matrix given policy \bmπ and \bmr_\bmπ is the reward signal given \bmπ. What annoys us is that P_\bmπ and \bmr_\bmπ are both mixed with \bmπ, which means every time when we update \bmπ, they will change together. In this paper, we leverage the notation from \citewang2007dual to disentangle \bmπ and environment dynamics which makes optimization over policy more straightforward. We show that policy gradient theorem \citesutton2018reinforcement and TRPO \citeschulman2015trust can be put into a more general framework and such notation has good potential to be extended to model-based reinforcement learning.

Cited by

• Adaptive Policy Backbone via Shared Network

Related

• Matrix Low-Rank Trust Region Policy Optimization
• Natural Policy Gradients In Reinforcement Learning Explained
• Model-free Policy Learning with Reward Gradients
• Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms
• Matrix Low-Rank Approximation For Policy Gradient Methods