POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning

2020/01/13 by Joseph Futoma, Michael C. Hughes, Futoma, Joseph +3 · 2 citations
Computer Science · Health Professions · Medicine · #Chronic Disease Management Strategies #FOS: Computer and information sciences #Healthcare Operations and Scheduling Optimization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2001.04032

openalex publication_date 2020/01/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many medical decision-making tasks can be framed as partially observed Markov decision processes (POMDPs). However, prevailing two-stage approaches that first learn a POMDP and then solve it often fail because the model that best fits the data may not be well suited for planning. We introduce a new optimization objective that (a) produces both high-performing policies and high-quality generative models, even when some observations are irrelevant for planning, and (b) does so in batch off-policy settings that are typical in healthcare, when only retrospective data is available. We demonstrate our approach on synthetic examples and a challenging medical decision-making problem.

Citations

• A tutorial on hidden Markov models and selected applications in speech recognition
• Planning and acting in partially observable stochastic domains
• Robust Policy Optimization with Baseline Guarantees
• The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making
• Predicting Blood Pressure Response to Fluid Bolus Therapy Using Attention-Based Neural Networks for Clinical Interpretability

Cited by

• Neural Index Policies for Restless Multi-Action Bandits with Heterogeneous Budgets
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI

Related

• Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
• The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• Semi-Markov Offline Reinforcement Learning for Healthcare
• Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs