Anti-Exploration by Random Network Distillation

2023/01/31 by Nikulin, Alexander, Kurenkov, Vladislav, Tarasov, Denis +1 · 7 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.2301.13616

Abstract

Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.

Cited by

• Scalable Offline Model-Based RL with Action Chunks
• ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• Noise-Guided Transport for Imitation Learning
• Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
• From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning
• Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data

Related

• Exploration and Anti-Exploration with Distributional Random Network Distillation
• Exploration by Random Network Distillation
• Exploration by Random Distribution Distillation
• SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction