Robust Bayesian reinforcement learning through tight lower bounds

2011/06/18 by Christos Dimitrakakis, Dimitrakakis, Christos
Computer Science · Decision Sciences · #Adaptive Dynamic Programming Control #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1106.3651

openalex publication_date 2011/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In the Bayesian approach to sequential decision making, exact calculation of the (subjective) utility is intractable. This extends to most special cases of interest, such as reinforcement learning problems. While utility bounds are known to exist for this problem, so far none of them were particularly tight. In this paper, we show how to efficiently calculate a lower bound, which corresponds to the utility of a near-optimal memoryless policy for the decision problem, which is generally different from both the Bayes-optimal policy and the policy which is optimal for the expected MDP under the current belief. We then show how these can be applied to obtain robust exploration policies in a Bayesian reinforcement learning setting.

Related

• A Greedy Approximation of Bayesian Reinforcement Learning with Probably Optimistic Transition Model
• Minimax-Bayes Reinforcement Learning
• Bayesian Residual Policy Optimization: Scalable Bayesian Reinforcement Learning with Clairvoyant Experts
• Bayesian Reinforcement Learning via Deep, Sparse Sampling
• Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling