2022/12/20 by Isaac J. Sledge, José C. Prı́ncipe, Sledge, Isaac J. +1
Computer Science · Economics, Econometrics and Finance · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Digital Games and Media #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Sports Analytics and Performance
paper · pdf · doi:10.48550/arxiv.2212.11083
openalex publication_date 2022/12/20 · openalex created_date 2023/01/04 · openalex updated_date 2026/07/28
In this paper, we consider the problem of adjusting the exploration rate when using value-of-information-based exploration. We do this by converting the value-of-information optimization into a problem of finding equilibria of a flow for a changing exploration rate. We then develop an efficient path-following scheme for converging to these equilibria and hence uncovering optimal action-selection policies. Under this scheme, the exploration rate is automatically adapted according to the agent's experiences. Global convergence is theoretically assured. We first evaluate our exploration-rate adaptation on the Nintendo GameBoy games Centipede and Millipede. We demonstrate aspects of the search process, like that it yields a hierarchy of state abstractions. We also show that our approach returns better policies in fewer episodes than conventional search strategies relying on heuristic, annealing-based exploration-rate adjustments. We then illustrate that these trends hold for deep, value-of-information-based agents that learn to play ten simple games and over forty more complicated games for the Nintendo GameBoy system. Performance either near or well above the level of human play is observed.