2022/04/06 by Tong Sang, Hongyao Tang, Sang, Tong +13
Computer Science · Engineering · Mathematics · Psychology · #Adaptation (eye) #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer science #Context (archaeology) #Decoupling (probability) #Engineering #FOS: Computer and information sciences #Generalization #Law #Machine Learning (cs.LG) #Machine learning #Mathematics #Online and offline #Political science #Psychology #Reinforcement Learning in Robotics #Reinforcement learning #Representation (politics) #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2204.02877
published in arXiv (Cornell University) (Cornell University) · Accepted on IJCAI 2022 and a previous version of this work is presented at the Generalizable Policy Learning in the Physical World Workshop (ICLR 2022)
openalex publication_date 2022/04/06 · arxiv created 2022/05/30 · arxiv updated 2022/05/31 · openalex created_date 2022/11/17 · openalex updated_date 2026/08/08
Deep Reinforcement Learning (DRL) has been a promising solution to many complex decision-making problems. Nevertheless, the notorious weakness in generalization among environments prevent widespread application of DRL agents in real-world scenarios. Although advances have been made recently, most prior works assume sufficient online interaction on training environments, which can be costly in practical cases. To this end, we focus on an offline-training-online-adaptation setting, in which the agent first learns from offline experiences collected in environments with different dynamics and then performs online policy adaptation in environments with new dynamics. In this paper, we propose Policy Adaptation with Decoupled Representations (PAnDR) for fast policy adaptation. In offline training phase, the environment representation and policy representation are learned through contrastive learning and policy recovery, respectively. The representations are further refined by mutual information optimization to make them more decoupled and complete. With learned representations, a Policy-Dynamics Value Function (PDVF) [Raileanu et al., 2020] network is trained to approximate the values for different combinations of policies and environments from offline experiences. In online adaptation phase, with the environment context inferred from few experiences collected in new environments, the policy is optimized by gradient ascent with respect to the PDVF. Our experiments show that PAnDR outperforms existing algorithms in several representative policy adaptation problems.