Learning to Explore with Meta-Policy Gradient

2018/03/13 by Tian-Bing Xu, Xu, Tianbing, Qiang Liu +5 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1803.05044

openalex publication_date 2018/03/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore local regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a global exploration that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning tasks.

Citations

• Deep Exploration via Bootstrapped DQN
• Learning to reinforcement learn
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Parameter Space Noise for Exploration

Cited by

• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning

Related

• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• Careful at Estimation and Bold at Exploration
• Curious Explorer: a provable exploration strategy in Policy Learning
• Meta Reinforcement Learning with Distribution of Exploration Parameters Learned by Evolution Strategies
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning