Perturbation-based exploration methods in deep reinforcement learning

2020/11/10 by Sneha Aenugu, Aenugu, Sneha
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Distributed Control Multi-Agent Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Modular Robots and Swarm Intelligence #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2011.05446

openalex publication_date 2020/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent research on structured exploration placed emphasis on identifying novel states in the state space and incentivizing the agent to revisit them through intrinsic reward bonuses. In this study, we question whether the performance boost demonstrated through these methods is indeed due to the discovery of structure in exploratory schedule of the agent or is the benefit largely attributed to the perturbations in the policy and reward space manifested in pursuit of structured exploration. In this study we investigate the effect of perturbations in policy and reward spaces on the exploratory behavior of the agent. We proceed to show that simple acts of perturbing the policy just before the softmax layer and introduction of sporadic reward bonuses into the domain can greatly enhance exploration in several domains of the arcade learning environment. In light of these findings, we recommend benchmarking any enhancements to structured exploration research against the backdrop of noisy exploration.

Citations

• Unifying Count-Based Exploration and Intrinsic Motivation
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement\n Learning
• Count-Based Exploration with Neural Density Models
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

Related

• Exploration by Random Reward Perturbation
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning\n Environment
• Fast active learning for pure exploration in reinforcement learning
• Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models