Generating Behaviorally Diverse Policies with Latent Diffusion Models

2023/05/30 by Hegde, Shashank, Batra, Sumeet, Zentner, K. R. +1 · 3 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2305.18738

Abstract

Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/home

Cited by

• Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
• Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
• From Parameters to Behavior: Unsupervised Compression of the Policy Space

Related

• Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
• Learning a Diffusion Model Policy from Rewards via Q-Score Matching
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces