Meta-Reinforcement Learning of Structured Exploration Strategies

2018/02/20 by Abhishek Gupta, Gupta, Abhishek, Russell Mendonca +7 · 7 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Advanced Multi-Objective Optimization Algorithms #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1802.07245

openalex publication_date 2018/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with structured noise (MAESN) -- to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.

Citations

• Continuous control with deep reinforcement learning
• Deep Exploration via Bootstrapped DQN
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Unifying Count-Based Exploration and Intrinsic Motivation
• Learning to reinforcement learn
• Prototypical Networks for Few-shot Learning
• Parameter Space Noise for Exploration
• Noisy Networks for Exploration
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• One-Shot Visual Imitation Learning via Meta-Learning

Cited by

• Efficient Reinforcement Learning for Zero-Shot Coordination in Evolving Games
• An Information-Theoretic Analysis of Out-of-Distribution Generalization in Meta-Learning with Applications to Meta-RL
• Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids
• Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning

Related

• Skill-based Meta-Reinforcement Learning
• MAME : Model-Agnostic Meta-Exploration
• Reinforcement Learning with a Focus on Adjusting Policies to Reach Targets
• Hypothesis Network Planned Exploration for Rapid Meta-Reinforcement Learning Adaptation
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning\n without Sacrifices