2023/02/23 by Gongpu Chen, Soung Chang Liew, Chen, Gongpu +1 · 2 citations
Computer Science · Engineering · #Age of Information Optimization #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Reinforcement Learning in Robotics #Smart Grid Energy Management #Systems and Control (eess.SY) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2302.11761
openalex publication_date 2023/02/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This paper investigates MDPs with intermittent state information. We consider a scenario where the controller perceives the state information of the process via an unreliable communication channel. The transmissions of state information over the whole time horizon are modeled as a Bernoulli lossy process. Hence, the problem is finding an optimal policy for selecting actions in the presence of state information losses. We first formulate the problem as a belief MDP to establish structural results. The effect of state information losses on the expected total discounted reward is studied systematically. Then, we reformulate the problem as a tree MDP whose state space is organized in a tree structure. Two finite-state approximations to the tree MDP are developed to find near-optimal policies efficiently. Finally, we put forth a nested value iteration algorithm for the finite-state approximations, which is proved to be faster than standard value iteration. Numerical results demonstrate the effectiveness of our methods.