Evolved Policy Gradients

2018/02/13 by Rein Houthooft, Houthooft, Rein, Richard Y. Chen +11 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1802.04821

openalex publication_date 2018/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a metalearning approach for learning gradient-based reinforcement learning (RL) algorithms. The idea is to evolve a differentiable loss function, such that an agent, which optimizes its policy to minimize this loss, will achieve high rewards. The loss is parametrized via temporal convolutions over the agent's experience. Because this loss is highly flexible in its ability to take into account the agent's history, it enables fast task learning. Empirical results show that our evolved policy gradient algorithm (EPG) achieves faster learning on several randomized environments compared to an off-the-shelf policy gradient method. We also demonstrate that EPG's learned loss can generalize to out-of-distribution test time tasks, and exhibits qualitatively different behavior from other popular metalearning algorithms.

Citations

• Continuous control with deep reinforcement learning
• Learning to Optimize
• Learning to reinforcement learn
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Count-Based Exploration with Neural Density Models
• One-Shot Visual Imitation Learning via Meta-Learning

Cited by

• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Population-Based Evolution Optimizes a Meta-Learning Objective
• Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks
• Gaussian Process Policy Optimization
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Multitask Soft Option Learning
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward

Related

• Gradient-based Reinforcement Planning in Policy-Search Methods
• Discovering Reinforcement Learning Algorithms
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Supplementing Gradient-Based Reinforcement Learning with Simple Evolutionary Ideas
• Policy Gradient for Reinforcement Learning with General Utilities