Wasserstein Distance Maximizing Intrinsic Control

2021/10/28 by Ishan Durugkar, Durugkar, Ishan, Steven Hansen +5 · 2 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2110.15331

openalex publication_date 2021/10/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper deals with the problem of learning a skill-conditioned policy that acts meaningfully in the absence of a reward signal. Mutual information based objectives have shown some success in learning skills that reach a diverse set of states in this setting. These objectives include a KL-divergence term, which is maximized by visiting distinct states even if those states are not far apart in the MDP. This paper presents an approach that rewards the agent for learning skills that maximize the Wasserstein distance of their state visitation from the start state of the skill. It shows that such an objective leads to a policy that covers more distance in the MDP than diversity based objectives, and validates the results on a variety of Atari environments.

Citations

• A Stochastic Approximation Method
• Stochastic Estimation of the Maximum of a Regression Function
• Variational Intrinsic Control
• Wasserstein Adversarial Imitation Learning
• What Can Learned Intrinsic Rewards Capture?
• Primal Wasserstein Imitation Learning
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Relative Variational Intrinsic Control

Cited by

• Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
• Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models

Related

• Wasserstein Unsupervised Reinforcement Learning
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching
• Exploration by Learning Diverse Skills through Successor State Measures