2020/05/16 by Ningyuan Chen, Chun Wang, Chen, Ningyuan +3 · 1 citation
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.2005.08088
openalex publication_date 2020/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
A standard assumption adopted in the multi-armed bandit (MAB) framework is that the mean rewards are constant over time. This assumption can be restrictive in the business world as decision-makers often face an evolving environment where the mean rewards are time-varying. In this paper, we consider a non-stationary MAB model with K arms whose mean rewards vary over time in a periodic manner. The unknown periods can be different across arms and scale with the length of the horizon T polynomially. We propose a two-stage policy that combines the Fourier analysis with a confidence-bound-based learning procedure to learn the periods and minimize the regret. In stage one, the policy correctly estimates the periods of all arms with high probability. In stage two, the policy explores the periodic mean rewards of arms using the periods estimated in stage one and exploits the optimal arm in the long run. We show that our learning policy incurs a regret upper bound O(√T∑k=1K Tk) where Tk is the period of arm k. Moreover, we establish a general lower bound Ω(√Tmaxk\ Tk\) for any policy. Therefore, our policy is near-optimal up to a factor of √(K).