Policy Gradient RL Algorithms as Directed Acyclic Graphs

2020/12/14 by Juan Jose Garau Luis, Luis, Juan Jose Garau
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Computer science #Directed acyclic graph #Directed graph #FOS: Computer and information sciences #Focus (optics) #Formal Methods in Verification #Machine Learning (cs.LG) #Mathematical optimization #Mathematics #Range (aeronautics) #Reinforcement Learning in Robotics #Reinforcement learning #Scheduling and Optimization Algorithms #Theoretical computer science #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.07763

published in ArXiv.org

openalex publication_date 2020/12/14 · arxiv created 2021/01/16 · arxiv updated 2021/01/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Meta Reinforcement Learning (RL) methods focus on automating the design of RL algorithms that generalize to a wide range of environments. The framework introduced in (Anonymous, 2020) addresses the problem by representing different RL algorithms as Directed Acyclic Graphs (DAGs), and using an evolutionary meta learner to modify these graphs and find good agent update rules. While the search language used to generate graphs in the paper serves to represent numerous already-existing RL algorithms (e.g., DQN, DDQN), it has limitations when it comes to representing Policy Gradient algorithms. In this work we try to close this gap by extending the original search language and proposing graphs for five different Policy Gradient algorithms: VPG, PPO, DDPG, TD3, and SAC.

Citations

• Human-level control through deep reinforcement learning
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Evolved Policy Gradients
• Addressing Function Approximation Error in Actor-Critic Methods
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Evolving Reinforcement Learning Algorithms

Related

• Evolving Reinforcement Learning Algorithms
• Discovering Reinforcement Learning Algorithms
• Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design
• LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration
• On the Convergence Theory of Meta Reinforcement Learning with Personalized Policies