Practical Probabilistic Model-based Deep Reinforcement Learning by Integrating Dropout Uncertainty and Trajectory Sampling

2023/09/20 by Wenjun Huang, Huang, Wenjun, Yunduan Cui +5
Computer Science · Engineering · #Muscle activation and electromyography studies #Prosthetics and Rehabilitation Robotics #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2309.11089

openalex publication_date 2023/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper addresses the prediction stability, prediction accuracy and control capability of the current probabilistic model-based reinforcement learning (MBRL) built on neural networks. A novel approach dropout-based probabilistic ensembles with trajectory sampling (DPETS) is proposed where the system uncertainty is stably predicted by combining the Monte-Carlo dropout and trajectory sampling in one framework. Its loss function is designed to correct the fitting error of neural networks for more accurate prediction of probabilistic models. The state propagation in its policy is extended to filter the aleatoric uncertainty for superior control capability. Evaluated by several Mujoco benchmark control tasks under additional disturbances and one practical robot arm manipulation task, DPETS outperforms related MBRL approaches in both average return and convergence velocity while achieving superior performance than well-known model-free baselines with significant sample efficiency. The open source code of DPETS is available at https://github.com/mrjun123/DPETS.

Related

• Dropout MPC: An Ensemble Neural MPC Approach for Systems with Learned Dynamics
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Synthesizing Neural Network Controllers with Probabilistic Model based Reinforcement Learning
• Bridging Active Exploration and Uncertainty-Aware Deployment Using Probabilistic Ensemble Neural Network Dynamics
• Mind the Uncertainty: Risk-Aware and Actively Exploring Model-Based Reinforcement Learning