2017/10/31 by Gregory Farquhar, Tim Rocktäschel, Farquhar, Gregory +5 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.1710.11417
openalex publication_date 2017/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Combining deep model-free reinforcement learning with on-line planning is a\npromising approach to building on the successes of deep RL. On-line planning\nwith look-ahead trees has proven successful in environments where transition\nmodels are known a priori. However, in complex environments where transition\nmodels need to be learned from data, the deficiencies of learned models have\nlimited their utility for planning. To address these challenges, we propose\nTreeQN, a differentiable, recursive, tree-structured model that serves as a\ndrop-in replacement for any value function network in deep RL with discrete\nactions. TreeQN dynamically constructs a tree by recursively applying a\ntransition model in a learned abstract state space and then aggregating\npredicted rewards and state-values using a tree backup to estimate Q-values. We\nalso propose ATreeC, an actor-critic variant that augments TreeQN with a\nsoftmax layer to form a stochastic policy network. Both approaches are trained\nend-to-end, such that the learned model is optimised for its actual use in the\ntree. We show that TreeQN and ATreeC outperform n-step DQN and A2C on a\nbox-pushing task, as well as n-step DQN and value prediction networks (Oh et\nal. 2017) on multiple Atari games. Furthermore, we present ablation studies\nthat demonstrate the effect of different auxiliary losses on learning\ntransition models.\n