2018/05/10 by Kenny Young, Young, Kenny, Baoxiang Wang +3
Computer Science · Decision Sciences · #Adaptive Dynamic Programming Control #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.1805.04514
openalex publication_date 2018/05/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Reinforcement learning (RL) has had many successes in both "deep" and\n"shallow" settings. In both cases, significant hyperparameter tuning is often\nrequired to achieve good performance. Furthermore, when nonlinear function\napproximation is used, non-stationarity in the state representation can lead to\nlearning instability. A variety of techniques exist to combat this --- most\nnotably large experience replay buffers or the use of multiple parallel actors.\nThese techniques come at the cost of moving away from the online RL problem as\nit is traditionally formulated (i.e., a single agent learning online without\nmaintaining a large database of training examples). Meta-learning can\npotentially help with both these issues by tuning hyperparameters online and\nallowing the algorithm to more robustly adjust to non-stationarity in a\nproblem. This paper applies meta-gradient descent to derive a set of step-size\ntuning algorithms specifically for online RL control with eligibility traces.\nOur novel technique, Metatrace, makes use of an eligibility trace analogous to\nmethods like TD(\λ). We explore tuning both a single scalar step-size\nand a separate step-size for each learned parameter. We evaluate Metatrace\nfirst for control with linear function approximation in the classic mountain\ncar problem and then in a noisy, non-stationary version. Finally, we apply\nMetatrace for control with nonlinear function approximation in 5 games in the\nArcade Learning Environment where we explore how it impacts learning speed and\nrobustness to initial step-size choice. Results show that the meta-step-size\nparameter of Metatrace is easy to set, Metatrace can speed learning, and\nMetatrace can allow an RL algorithm to deal with non-stationarity in the\nlearning task.\n