2005/01/01 by Nicholas Roy, N. Roy, Geoffrey J. Gordon +3 · 255 citations
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Bayesian Modeling and Causal Inference #Bellman equation #Computer science #Curse of dimensionality #Function (biology) #Function approximation #Machine Learning and Algorithms #Machine learning #Markov chain #Markov decision process #Markov model #Markov process #Mathematical optimization #Mathematics #Optimal control #Partially observable Markov decision process #Reinforcement Learning in Robotics #Space (punctuation) #State space #Subspace topology #cs.AI
paper · pdf · doi:10.1613/jair.1496
published in Journal of Artificial Intelligence Research 23, 1-40 (AI Access Foundation)
openalex publication_date 2005/01/01 · arxiv created 2011/10/04 · arxiv updated 2011/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Standard value function approaches to finding policies for Partially Observable Markov Decision Processes (POMDPs) are generally considered to be intractable for large models. The intractability of these algorithms is to a large extent a consequence of computing an exact, optimal policy over the entire belief space. However, in real-world POMDP problems, computing the optimal policy for the full belief space is often unnecessary for good control even for problems with complicated policy classes. The beliefs experienced by the controller often lie near a structured, low-dimensional subspace embedded in the high-dimensional belief space. Finding a good approximation to the optimal value function for only this subspace can be much easier than computing the full value function. We introduce a new method for solving large-scale POMDPs by reducing the dimensionality of the belief space. We use Exponential family Principal Components Analysis (Collins, Dasgupta & Schapire, 2002) to represent sparse, high-dimensional belief spaces using small sets of learned features of the belief state. We then plan only in terms of the low-dimensional belief features. By planning in this low-dimensional space, we can find policies for POMDP models that are orders of magnitude larger than models that can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and on mobile robot navigation tasks.