Solving POMDPs by Searching in Policy Space

2013/01/30 by Eric A. Hansen, Hansen, Eric A. · 2 citations
Computer Science · Engineering · #Advanced Control Systems Optimization #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Formal Methods in Verification #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1301.7380

openalex publication_date 2013/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most algorithms for solving POMDPs iteratively improve a value function that implicitly represents a policy and are said to search in value function space. This paper presents an approach to solving POMDPs that represents a policy explicitly as a finite-state controller and iteratively improves the controller by search in policy space. Two related algorithms illustrate this approach. The first is a policy iteration algorithm that can outperform value iteration in solving infinitehorizon POMDPs. It provides the foundation for a new heuristic search algorithm that promises further speedup by focusing computational effort on regions of the problem space that are reachable, or likely to be reached, from a start state.

Citations

• The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
• Planning and acting in partially observable stochastic domains

Cited by

• Memoryless Policy Iteration for Episodic POMDPs
• Partially Observable Monte-Carlo Graph Search
• Partially Observable Markov Decision Processes with Behavioral Norms
• The Complexity of Decentralized Control of Markov Decision Processes

Related

• Heuristic Search Value Iteration for POMDPs
• Solving POMDPs by Searching the Space of Finite Policies
• My Brain is Full: When More Memory Helps
• Approximate Dec-POMDP Solving Using Multi-Agent A*
• Point-Based POMDP Algorithms: Improved Analysis and Implementation