Goal-Conditioned Generators of Deep Policies

2022/07/04 by Faccio, Francesco, Herrmann, Vincent, Ramesh, Aditya +2 · 2 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2207.01570

Abstract

Goal-conditioned Reinforcement Learning (RL) aims at learning optimal policies, given goals encoded in special command inputs. Here we study goal-conditioned neural nets (NNs) that learn to generate deep NN policies in form of context-specific weight matrices, similar to Fast Weight Programmers and other methods from the 1990s. Using context commands of the form "generate a policy that achieves a desired expected return," our NN generators combine powerful exploration of parameter space with generalization across commands to iteratively find better and better policies. A form of weight-sharing HyperNetworks and policy embeddings scales our method to generate deep NNs. Experiments show how a single learned policy generator can produce policies that achieve any return seen during training. Finally, we evaluate our algorithm on a set of continuous control tasks where it exhibits competitive performance. Our code is public.

Cited by

• HyperVLA: Efficient Inference in Vision-Language-Action Models via Hypernetworks
• From Parameters to Behavior: Unsupervised Compression of the Policy Space

Related

• Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning
• Upside Down Reinforcement Learning with Policy Generators
• Goal-Conditioned Reinforcement Learning: Problems and Solutions
• Learning to Synthesize Programs as Interpretable and Generalizable\n Policies
• DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policies