One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

2022/11/30 by Marc Rigter, Bruno Lacerda, Rigter, Marc +3 · 5 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2212.00124

openalex publication_date 2022/11/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Offline reinforcement learning (RL) is suitable for safety-critical domains where online exploration is too costly or dangerous. In such safety-critical settings, decision-making should take into consideration the risk of catastrophic outcomes. In other words, decision-making should be risk-sensitive. Previous works on risk in offline RL combine together offline RL techniques, to avoid distributional shift, with risk-sensitive RL algorithms, to achieve risk-sensitivity. In this work, we propose risk-sensitivity as a mechanism to jointly address both of these issues. Our model-based approach is risk-averse to both epistemic and aleatoric uncertainty. Risk-aversion to epistemic uncertainty prevents distributional shift, as areas not covered by the dataset have high epistemic uncertainty. Risk-aversion to aleatoric uncertainty discourages actions that may result in poor outcomes due to environment stochasticity. Our experiments show that our algorithm achieves competitive performance on deterministic benchmarks, and outperforms existing approaches for risk-sensitive objectives in stochastic domains.

Cited by

• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
• Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
• Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model
• Hybrid Cross-domain Robust Reinforcement Learning

Related

• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
• Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
• Risk-Averse Offline Reinforcement Learning
• Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
• Wall Street Tree Search: Risk-Aware Planning for Offline Reinforcement Learning