Adapting the Function Approximation Architecture in Online Reinforcement\n Learning

2021/06/17 by John D. Martin, Joseph Modayil, Martin, John D. +1
Computer Science · #Neural Networks and Reservoir Computing

paper · pdf · doi:10.48550/arxiv.2106.09776

Abstract

The performance of a reinforcement learning (RL) system depends on the\ncomputational architecture used to approximate a value function. Deep learning\nmethods provide both optimization techniques and architectures for\napproximating nonlinear functions from noisy, high-dimensional observations.\nHowever, prevailing optimization techniques are not designed for\nstrictly-incremental online updates. Nor are standard architectures designed\nfor observations with an a priori unknown structure: for example, light sensors\nrandomly dispersed in space. This paper proposes an online RL prediction\nalgorithm with an adaptive architecture that efficiently finds useful nonlinear\nfeatures. The algorithm is evaluated in a spatial domain with high-dimensional,\nstochastic observations. The algorithm outperforms non-adaptive baseline\narchitectures and approaches the performance of an architecture given\nside-channel information. These results are a step towards scalable RL\nalgorithms for more general problems, where the observation structure is not\navailable.\n

Citations

• Gradient-based learning applied to document recognition

Related

• Towards model-free RL algorithms that scale well with unstructured data
• Online Model Selection for Reinforcement Learning with Function Approximation
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Adaptive Resolving Methods for Reinforcement Learning with Function Approximations