Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making

2020/02/05 by Chengchun Shi, Runzhe Wan, Shi, Chengchun +7 · 9 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · #Advanced Causal Inference Techniques #Artificial intelligence #Computer science #Gene Regulatory Network Analysis #Machine learning #Markov chain #Markov decision process #Markov model #Markov process #Markov property #Mathematics #Parametric statistics #Partially observable Markov decision process #Property (philosophy) #Reinforcement Learning in Robotics #Reinforcement learning #Statistics #Variable-order Markov model #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.01751

published in arXiv (Cornell University) (Cornell University)

arxiv created 2020/02/05 · openalex publication_date 2020/02/05 · arxiv updated 2020/02/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an important role for identifying the optimal policy in high-order Markov decision processes and partially observable MDPs. We apply our test to both synthetic datasets and a real data example from mobile health studies to illustrate its usefulness.

Cited by

• Capturing reduced-order quantum many-body dynamics out of equilibrium via neural ordinary differential equations
• Off-Policy Evaluation Under Nonignorable Missing Data
• Tabular reinforcement learning for reward robust, explainable crop rotation policies matching deep reinforcement learning performance
• Prediction-Based Markov Violation Scores for Detecting Non-Markovian Observations in Reinforcement Learning

Related

• Consistent Order Determination of Markov Decision Process
• A Bayesian Approach to Learning Bandit Structure in Markov Decision Processes
• Sequential Decision-Making under Uncertainty: A Robust MDPs review
• Reinforcement Learning: Stochastic Approximation Algorithms for Markov Decision Processes
• Reinforcement Learning When All Actions are Not Always Available