Posterior sampling for reinforcement learning: worst-case regret bounds

2017/05/19 by Agrawal, Shipra, Jia, Randy
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1705.07041

Abstract

We present an algorithm based on posterior sampling (aka Thompson sampling) that achieves near-optimal worst-case regret bounds when the underlying Markov Decision Process (MDP) is communicating with a finite, though unknown, diameter. Our main result is a high probability regret upper bound of O(DS√(AT)) for any communicating MDP with S states, A actions and diameter D. Here, regret compares the total reward achieved by the algorithm to the total expected reward of an optimal infinite-horizon undiscounted average reward policy, in time horizon T. This result closely matches the known lower bound of Ω(√(DSAT)). Our techniques involve proving some novel results about the anti-concentration of Dirichlet distribution, which may be of independent interest.

Related

• Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling
• Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees
• Provably Efficient Exploration in Constrained Reinforcement Learning:Posterior Sampling Is All You Need
• Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPs
• (More) Efficient Reinforcement Learning via Posterior Sampling