2019/02/04 by Ari Arapostathis, Arapostathis, Ari, Vivek S. Borkar +1 · 1 citation
Computer Science · Economics, Econometrics and Finance · Mathematics · #90C40 (93E20) #FOS: Electrical engineering #FOS: Mathematics #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Stochastic processes and financial applications #Systems and Control (eess.SY) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1902.01048
openalex publication_date 2019/02/04 · openalex created_date 2022/10/03 · openalex updated_date 2026/08/01
We study Markov decision processes with Polish state and action spaces. The\naction space is state dependent and is not necessarily compact. We first\nestablish the existence of an optimal ergodic occupation measure using only a\nnear-monotone hypothesis on the running cost. Then we study the well-posedness\nof Bellman equation, or what is commonly known as the average cost optimality\nequation, under the additional hypothesis of the existence of a small set. We\ndeviate from the usual approach which is based on the vanishing discount method\nand instead map the problem to an equivalent one for a controlled split chain.\nWe employ a stochastic representation of the Poisson equation to derive the\nBellman equation. Next, under suitable assumptions, we establish convergence\nresults for the 'relative value iteration' algorithm which computes the\nsolution of the Bellman equation recursively. In addition, we present some\nresults concerning the stability and asymptotic optimality of the associated\nrolling horizon policies.\n