2022/04/05 by Sarah Bechtle, Bechtle, Sarah, Ludovic Righetti +3
Physics and Astronomy · Computer Science · #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Domain Adaptation and Few-Shot Learning
paper · pdf · doi:10.48550/arxiv.2204.02210
Being able to seamlessly generalize across different tasks is fundamental for robots to act in our world. However, learning representations that generalize quickly to new scenarios is still an open research problem in reinforcement learning. In this paper we present a framework to meta-learn the critic for gradient-based policy learning. Concretely, we propose a model-based bi-level optimization algorithm that updates the critics parameters such that the policy that is learned with the updated critic gets closer to solving the meta-training tasks. We illustrate that our algorithm leads to learned critics that resemble the ground truth Q function for a given task. Finally, after meta-training, the learned critic can be used to learn new policies for new unseen task and environment settings via model-free policy gradient optimization, without requiring a model. We present results that show the generalization capabilities of our learned critic to new tasks and dynamics when used to learn a new policy in a new scenario.