Reachable Space Characterization of Markov Decision Processes with Time Variability

2019/05/22 by Junhong Xu, Kai Yin, Xu, Junhong +3
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Fault Detection and Control Systems #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.1905.09342

openalex publication_date 2019/05/22 · openalex created_date 2019/05/29 · openalex updated_date 2026/07/28

Abstract

We propose a solution to a time-varying variant of Markov Decision Processes which can be used to address decision-theoretic planning problems for autonomous systems operating in unstructured outdoor environments. We explore the time variability property of the planning stochasticity and investigate the state reachability, based on which we then develop an efficient iterative method that offers a good trade-off between solution optimality and time complexity. The reachability space is constructed by analyzing the means and variances of states' reaching time in the future. We validate our algorithm through extensive simulations using ocean data, and the results show that our method achieves a great performance in terms of both solution quality and computing time.

Citations

• Reinforcement learning in robotics: A survey
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Reinforcement learning: The Good, The Bad and The Ugly

Related

• A Solution to Time-Varying Markov Decision Processes
• Solving Markov Decision Processes with Reachability Characterization\n from Mean First Passage Times
• Weathering Ongoing Uncertainty: Learning and Planning in a Time-Varying Partially Observable Environment
• Continuous-Time Markov Decisions based on Partial Exploration
• Planning under periodic observations: bounds and bounding-based solutions