Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning

2020/04/22 by Shangtong Zhang, Bo Liu, Zhang, Shangtong +3 · 4 citations
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neurological disorders and treatments #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2004.10888

openalex publication_date 2020/04/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.

Citations

• On the Finite Time Convergence of Cyclic Coordinate Descent Methods
• Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization
• Trust Region Policy Optimization
• Algorithms for CVaR Optimization in MDPs
• Continuous control with deep reinforcement learning
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Consistent On-Line Off-Policy Evaluation
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Addressing Function Approximation Error in Actor-Critic Methods
• Meta-Gradient Reinforcement Learning
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• GenDICE: Generalized Offline Estimation of Stationary Values
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation

Cited by

• Sharpe Ratio Optimization in Markov Decision Processes
• Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
• Risk-Averse Total-Reward Reinforcement Learning
• Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions

Related

• An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient
• Mean-Variance Efficient Reinforcement Learning with Applications to Dynamic Financial Investment
• Sharp Variance-Dependent Bounds in Reinforcement Learning: Best of Both Worlds in Stochastic and Deterministic Environments
• Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning
• Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods