Adversarial Intrinsic Motivation for Reinforcement Learning

2021/05/27 by Ishan Durugkar, Durugkar, Ishan, Mauricio Tec +5 · 8 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2105.13345

openalex publication_date 2021/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectively for reinforcement learning (RL) tasks. Specifically, this paper focuses on goal-conditioned reinforcement learning where the idealized (unachievable) target distribution has full measure at the goal. This paper introduces a quasimetric specific to Markov Decision Processes (MDPs) and uses this quasimetric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in as few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function. Our experiments show that this reward function changes smoothly with respect to transitions in the MDP and directs the agent's exploration to find the goal efficiently. Additionally, we combine AIM with Hindsight Experience Replay (HER) and show that the resulting algorithm accelerates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.

Citations

• Reinforcement Learning with Deep Energy-Based Policies
• Hindsight Experience Replay
• Provably Efficient Maximum Entropy Exploration
• Wasserstein Adversarial Imitation Learning
• Efficient Exploration via State Marginal Matching
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Automatic Curriculum Learning through Value Disagreement
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable\n Environments

Cited by

• Action-Sufficient Goal Representations
• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• Inter-Agent Relative Representations for Multi-Agent Option Discovery
• Efficient Skill Discovery via Regret-Aware Optimization
• State-Covering Trajectory Stitching for Diffusion Planners
• Wasserstein Distance Maximizing Intrinsic Control
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning

Related

• Wasserstein Adversarial Imitation Learning
• Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning
• Goal-Conditioned Imitation Learning
• Automatic Goal Generation using Dynamical Distance Learning
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching