2021/01/06 by Kaige Yang, Yang, Kaige
Computer Science · Neuroscience · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #EEG and Brain-Computer Interfaces #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.2101.02230
openalex publication_date 2021/01/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Transfer reinforcement learning aims to improve the sample efficiency of solving unseen new tasks by leveraging experiences obtained from previous tasks. We consider the setting where all tasks (MDPs) share the same environment dynamic except reward function. In this setting, the MDP dynamic is a good knowledge to transfer, which can be inferred by uniformly random policy. However, trajectories generated by uniform random policy are not useful for policy improvement, which impairs the sample efficiency severely. Instead, we observe that the binary MDP dynamic can be inferred from trajectories of any policy which avoids the need of uniform random policy. As the binary MDP dynamic contains the state structure shared over all tasks we believe it is suitable to transfer. Built on this observation, we introduce a method to infer the binary MDP dynamic on-line and at the same time utilize it to guide state embedding learning, which is then transferred to new tasks. We keep state embedding learning and policy learning separately. As a result, the learned state embedding is task and policy agnostic which makes it ideal for transfer learning. In addition, to facilitate the exploration over the state space, we propose a novel intrinsic reward based on the inferred binary MDP dynamic. Our method can be used out-of-box in combination with model-free RL algorithms. We show two instances on the basis of \algoDQN and \algoA2C. Empirical results of intensive experiments show the advantage of our proposed method in various transfer learning tasks.