2024/06/04 by Nir Greshler, Greshler, Nir, David Ben Eli +11 · 1 citation
Computer Science · Psychology · Social Sciences · #Artificial Intelligence (cs.AI) #Educational Games and Gamification #FOS: Computer and information sciences #Teaching and Learning Programming #Wikis in Education and Collaboration
paper · pdf · doi:10.48550/arxiv.2406.02103
openalex publication_date 2024/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The combination of Monte Carlo tree search and neural networks has revolutionized online planning. As neural network approximations are often imperfect, we ask whether uncertainty estimates about the network outputs could be used to improve planning. We develop a Bayesian planning approach that facilitates such uncertainty quantification, inspired by classical ideas from the meta-reasoning literature. We propose a Thompson sampling based algorithm for searching the tree of possible actions, for which we prove the first (to our knowledge) finite time Bayesian regret bound, and propose an efficient implementation for a restricted family of posterior distributions. In addition we propose a variant of the Bayes-UCB method applied to trees. Empirically, we demonstrate that on the ProcGen Maze and Leaper environments, when the uncertainty estimates are accurate but the neural network output is inaccurate, our Bayesian approach searches the tree much more effectively. In addition, we investigate whether popular uncertainty estimation methods are accurate enough to yield significant gains in planning. Our code is available at: https://github.com/nirgreshler/bayesian-online-planning.