Scaling Internal-State Policy-Gradient Methods for POMDPs

2025/12/02 by Douglas Aberdeen, Aberdeen, Douglas, Jonathan Baxter +1
Computer Science · Decision Sciences · #Reinforcement Learning in Robotics #Advanced Bandit Algorithms Research #Adaptive Dynamic Programming Control

paper · pdf · doi:10.48550/arxiv.2512.03204

Abstract

Policy-gradient methods have received increased attention recently as a mechanism for learning to act in partially observable environments. They have shown promise for problems admitting memoryless policies but have been less successful when memory is required. In this paper we develop several improved algorithms for learning policies with memory in an infinite-horizon setting -- directly when a known model of the environment is available, and via simulation otherwise. We compare these algorithms on some large POMDPs, including noisy robot navigation and multi-agent problems.

Citations

• Policy Improvement for POMDPs Using Normalized Importance Sampling
• Solving POMDPs by Searching the Space of Finite Policies
• The Complexity of Markov Decision Processes

Related

• Memory-based Deep Reinforcement Learning for POMDPs
• Robust Finite-Memory Policy Gradients for Hidden-Model POMDPs
• Policy Gradient in Partially Observable Environments: Approximation and Convergence
• My Brain is Full: When More Memory Helps
• Learning Policies with External Memory