Better Optimism By Bayes: Adaptive Planning with Rich Models

2014/02/09 by Arthur Guez, David Silver, Guez, Arthur +3
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1402.1958

openalex publication_date 2014/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The computational costs of inference and planning have confined Bayesian model-based reinforcement learning to one of two dismal fates: powerful Bayes-adaptive planning but only for simplistic models, or powerful, Bayesian non-parametric models but using simple, myopic planning strategies such as Thompson sampling. We ask whether it is feasible and truly beneficial to combine rich probabilistic models with a closer approximation to fully Bayesian planning. First, we use a collection of counterexamples to show formal problems with the over-optimism inherent in Thompson sampling. Then we leverage state-of-the-art techniques in efficient Bayes-adaptive planning and non-parametric Bayesian methods to perform qualitatively better than both existing conventional algorithms and Thompson sampling on two contextual bandit-like problems.

Citations

• Model-Based Bayesian Reinforcement Learning in Large Structured Domains
• Bayesian Density Estimation and Inference Using Mixtures

Related

• Bayesian Reinforcement Learning via Deep, Sparse Sampling
• A Greedy Approximation of Bayesian Reinforcement Learning with Probably Optimistic Transition Model
• Bayesian bandits: balancing the exploration-exploitation tradeoff via double sampling
• Learning is planning: near Bayes-optimal reinforcement learning via\n Monte-Carlo tree search
• Variational Bayesian Optimistic Sampling