Comparing discounted and average-cost Markov Decision Processes: a statistical significance perspective

2021/12/01 by Solms, Dylan
#Applications (stat.AP) #FOS: Computer and information sciences #FOS: Mathematics #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2112.00684

Abstract

Optimal Markov Decision Process policies for problems with finite state and action space are identified through a partial ordering by comparing the value function across states. This is referred to as state-based optimality. This paper identifies when such optimality guarantees some form of system-based optimality as measured by a scalar. Four such system-based metrics are introduced. Uni-variate empirical distributions of these metrics are obtained through simulation as to assess whether theoretically optimal policies provide a statistically significant advantage. This has been conducted using a Student's t-test, Welch's t-test and a Mann-Whitney U-test. The proposed method is applied to a common problem in queuing theory: admission control.

Related

• On the Optimality Equation for Average Cost Markov Decision Processes and its Validity for Inventory Control
• Average Cost Markov Decision Processes with Semi-Uniform Feller Transition Probabilities
• Average-Cost MDPs with Infinite State and Action Sets: New Sufficient Conditions for Optimality Inequalities and Equations
• On the Reduction of Total-Cost and Average-Cost MDPs to Discounted MDPs
• An axiomatic approach to Markov decision processes