Recurrent Natural Policy Gradient for POMDPs

2024/05/28 by Cayci, Semih, Eryilmaz, Atilla · 3 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2405.18221

Abstract

Solving partially observable Markov decision processes (POMDPs) remains a fundamental challenge in reinforcement learning (RL), primarily due to the curse of dimensionality induced by the non-stationarity of optimal policies. In this work, we study a natural actor-critic (NAC) algorithm that integrates recurrent neural network (RNN) architectures into a natural policy gradient (NPG) method and a temporal difference (TD) learning method. This framework leverages the representational capacity of RNNs to address non-stationarity in RL to solve POMDPs while retaining the statistical and computational efficiency of natural gradient methods in RL. We provide non-asymptotic theoretical guarantees for this method, including bounds on sample and iteration complexity to achieve global optimality up to function approximation. Additionally, we characterize pathological cases that stem from long-term dependencies, thereby explaining limitations of RNN-based policy optimization for POMDPs.

Cited by

• Scalable Policy-Based RL Algorithms for POMDPs
• Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
• Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

Related

• Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges
• Natural Policy Gradient for Average Reward Non-Stationary RL
• Neural Network Compatible Off-Policy Natural Actor-Critic Algorithm
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
• Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems