Distributed Distributional Deterministic Policy Gradients

2018/04/23 by Gabriel Barth-Maron, Barth-Maron, Gabriel, Matthew W. Hoffman +15 · 16 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotic Locomotion and Control #Zebrafish Biomedical Research Applications

paper · pdf · doi:10.48550/arxiv.1804.08617

openalex publication_date 2018/04/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of N-step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.

Citations

• On the Theory of the Brownian Motion
• Continuous control with deep reinforcement learning

Cited by

• Generative Actor Critic
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks
• Unleashing Flow Policies with Distributional Critics
• Learning to Solve a Rubik's Cube with a Dexterous Hand
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Never Give Up: Learning Directed Exploration Strategies
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Categorical Policies: Multimodal Policy Learning and Exploration in Continuous Control
• Task-Agnostic Morphology Evolution
• One-Step Flow Policy Mirror Descent
• Positive-Unlabeled Reward Learning
• Generative Adversarial Imagination for Sample Efficient Deep\n Reinforcement Learning

Related

• Sample-based Distributional Policy Gradient
• Continuous Control Reinforcement Learning: Distributed Distributional DrQ Algorithms
• Deterministic Value-Policy Gradients
• Deterministic Policy Gradients With General State Transitions
• Policy Search by Target Distribution Learning for Continuous Control