Approximate Value Iteration for Risk-aware Markov Decision Processes

2017/01/05 by P. L. Yu, William B. Haskell, Yu, Pengqian +3 · 2 citations
Decision Sciences · Mathematics · #FOS: Electrical engineering #FOS: Mathematics #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Probabilistic and Robust Engineering Design #Risk and Portfolio Optimization #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1701.01290

openalex publication_date 2017/01/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider large-scale Markov decision processes (MDPs) with a risk measure of variability in cost, under the risk-aware MDPs paradigm. Previous studies showed that risk-aware MDPs, based on a minimax approach to handling risk, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimensionality", MDPs that model real-life problems are typically prohibitively large for such approaches. In this paper, we employ an approximate dynamic programming approach, and develop a family of simulation-based algorithms to approximately solve large-scale risk-aware MDPs. In parallel, we develop a unified convergence analysis technique to derive sample complexity bounds for this new family of algorithms.

Cited by

• Online Policy Evaluation for MDPs with Dynamic UBSR Measures
• Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents

Related

• Scaling Up Robust MDPs by Reinforcement Learning
• Planning and Learning in Average Risk-aware MDPs
• On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes
• An Approximate Solution Method for Large Risk-Averse Markov Decision\n Processes
• Approximate Dynamic Programming By Minimizing Distributionally Robust\n Bounds