Robust Markov Decision Processes

2012/11/15 by Wolfram Wiesemann, Daniel Kuhn, Daniel Kühn +2 · 33 citations
Computer Science · Decision Sciences · Mathematics · #Bayesian Modeling and Causal Inference #Fuzzy Systems and Optimization #Risk and Portfolio Optimization

paper · doi:10.1287/moor.1120.0566

openalex publication_date 2012/11/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Markov decision processes (MDPs) are powerful tools for decision making in uncertain dynamic environments. However, the solutions of MDPs are of limited practical use because of their sensitivity to distributional model parameters, which are typically unknown and have to be estimated by the decision maker. To counter the detrimental effects of estimation errors, we consider robust MDPs that offer probabilistic guarantees in view of the unknown parameters. To this end, we assume that an observation history of the MDP is available. Based on this history, we derive a confidence region that contains the unknown parameters with a prespecified probability 1-β. Afterward, we determine a policy that attains the highest worst-case performance over this confidence region. By construction, this policy achieves or exceeds its worst-case performance with a confidence of at least 1-β. Our method involves the solution of tractable conic programs of moderate size.

Citations

• Robust Dynamic Programming
• Semidefinite Programming

Cited by

• Online Robust Reinforcement Learning with General Function Approximation
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• Bayesian Ambiguity Contraction-based Adaptive Robust Markov Decision Processes for Adversarial Surveillance Missions
• On the Tension Between Optimality and Adversarial Robustness in Policy Optimization
• RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
• Provably Efficient Sample Complexity for Robust CMDP
• Distributionally Robust Self Paced Curriculum Reinforcement Learning
• Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPs
• Robust mean-field control under common noise uncertainty
• Computational Hardness of Static Distributionally Robust Markov Decision Processes
• On Structural Properties of Risk-Averse Optimal Stopping Problems
• Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets
• Deep SPI: Safe Policy Improvement via World Models
• ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty
• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
• MDP modeling for multi-stage stochastic programs
• Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Online Bayesian Risk-Averse Reinforcement Learning
• Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
• Simple Optimizers for Convex Aligned Multi-Objective Optimization
• Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
• Stability-certified reinforcement learning: A control-theoretic perspective
• Best-Effort Policies for Robust Markov Decision Processes
• Convergence of Fast Policy Iteration in Markov Games and Robust MDPs
• Probabilistic Alternating Simulations for Policy Synthesis in Uncertain Stochastic Dynamical Systems
• ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning
• Extended Radial Basis Function Controller for Reinforcement Learning
• Efficient Solution and Learning of Robust Factored MDPs
• Robust Policy Optimization with Baseline Guarantees

Related

• Distributionally Robust Markov Decision Processes
• Distributionally Robust Counterpart in Markov Decision Processes
• Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
• Robust Markov Decision Processes without Model Estimation
• Solving Robust Markov Decision Processes: Generic, Reliable, Efficient