Dropout Q-Functions for Doubly Efficient Reinforcement Learning

2021/10/05 by Takuya Hiraoka, T. HIRAOKA, Hiraoka, Takuya +8 · 23 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2110.02034

ICLR 2022. Source code: https://github.com/TakuyaHiraoka/Dropout-Q-Functions-for-Doubly-Efficient-Reinforcement-Learning Poster: https://drive.google.com/file/d/1_JSuwlUsMjzo6zRaAIcXXj3__AmOvu2t/view?usp=sharing Slides: https://drive.google.com/file/d/1ecq9SQ2KSNpfeblCkr6TYPz5gRk_Y4S8/view?usp=sharing

openalex publication_date 2021/10/05 · arxiv created 2022/03/16 · arxiv updated 2022/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Continuous control with deep reinforcement learning
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Understanding and Improving Layer Normalization
• Learning dexterous in-hand manipulation
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning
• Acme: A Research Framework for Distributed Reinforcement Learning
• Meta-Model-Based Meta-Policy Optimization
• Bidirectional Model-based Policy Optimization
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

Cited by

• Double Horizon Model-Based Policy Optimization
• Fast Policy Learning for 6-DOF Position Control of Underwater Vehicles
• Dexterous Robotic Piano Playing at Scale
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• Geometric Properties of Neural Multivariate Regression
• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
• An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
• Real-time reinforcement learning for turbulent state-dependent control in a bluff-body wake
• Solving Robotics Tasks with Prior Demonstration via Exploration-Efficient Deep Reinforcement Learning
• Reinforcement learning entangling operations on spin qubits
• A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• Online Training and Pruning of Deep Reinforcement Learning Networks
• Exploring the robustness of TractOracle methods in RL-based tractography
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
• The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
• Scaling Algorithm Distillation for Continuous Control with Mamba
• Scaling CrossQ with Weight Normalization
• Growable and Interpretable Neural Control with Online Continual Learning for Autonomous Lifelong Locomotion Learning Machines
• ReBRAC-v2: The Return of the King
• Behavior-Consistent Deep Reinforcement Learning
• Reward-Conditioned Reinforcement Learning
• Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss

Related

• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• On the Reduction of Variance and Overestimation of Deep Q-Learning
• Ensemble Bootstrapping for Q-Learning
• Deep Double Q-learning