Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel

2022/10/08 by Dahlin, Nathan, Bose, Subhonmesh, Veeravalli, Venugopal V.
#FOS: Electrical engineering #FOS: Mathematics #Optimization and Control (math.OC) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2210.04098

Abstract

We consider the control of a Markov decision process (MDP) that undergoes an abrupt change in its transition kernel (mode). We formulate the problem of minimizing regret under control-switching based on mode change detection, compared to a mode-observing controller, as an optimal stopping problem. Using a sequence of approximations, we reduce it to a quickest change detection (QCD) problem with Markovian data, for which we characterize a state-dependent threshold-type optimal change detection policy. Numerical experiments illustrate various properties of our control-switching policy.

Related

• Quickest Change Detection Approach to Optimal Control in Markov Decision Processes with Model Changes
• Cascade Markov Decision Processes: Theory and Applications
• Finite-Horizon Markov Decision Processes with Sequentially-Observed Transitions
• Stabilization of regime-switching processes by feedback control based on discrete time observations II: state-dependent case
• A Sliding-Window Algorithm for Markov Decision Processes with\n Arbitrarily Changing Rewards and Transitions