Estimating Risk and Uncertainty in Deep Reinforcement Learning

2019/05/23 by William R. Clements, Bastien Van Delft, Clements, William R. +7 · 3 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1905.09638

openalex publication_date 2019/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning agents are faced with two types of uncertainty. Epistemic uncertainty stems from limited data and is useful for exploration, whereas aleatoric uncertainty arises from stochastic environments and must be accounted for in risk-sensitive applications. We highlight the challenges involved in simultaneously estimating both of them, and propose a framework for disentangling and estimating these uncertainties on learned Q-values. We derive unbiased estimators of these uncertainties and introduce an uncertainty-aware DQN algorithm, which we show exhibits safe learning behavior and outperforms other DQN variants on the MinAtar testbed.

Citations

• Bayesian quantile regression
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement\n Learning
• AI Safety Gridworlds
• Distributed Distributional Deterministic Policy Gradients
• Information-Directed Exploration for Deep Reinforcement Learning
• Weight Uncertainty in Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Cited by

• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
• Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learning
• Auditing the Risk Claims of Distributional Reinforcement Learning

Related

• Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning
• A Review of Uncertainty for Deep Reinforcement Learning
• Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning
• Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learning
• Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning