Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs

2019/05/17 by Luchen Li, Matthieu Komorowski, Li, Luchen +3 · 2 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Sepsis Diagnosis and Treatment

paper · pdf · doi:10.48550/arxiv.1905.07465

openalex publication_date 2019/05/17 · openalex created_date 2019/05/29 · openalex updated_date 2026/07/28

Abstract

Health-related data is noisy and stochastic in implying the true physiological states of patients, limiting information contained in single-moment observations for sequential clinical decision making. We model patient-clinician interactions as partially observable Markov decision processes (POMDPs) and optimize sequential treatment based on belief states inferred from history sequence. To facilitate inference, we build a variational generative model and boost state representation with a recurrent neural network (RNN), incorporating an auxiliary loss from sequence auto-encoding. Meanwhile, we optimize a continuous policy of drug levels with an actor-critic method where policy gradients are obtained from a stablized off-policy estimate of advantage function, with the value of belief state backed up by parallel best-first suffix trees. We exploit our methodology in optimizing dosages of vasopressor and intravenous fluid for sepsis patients using a retrospective intensive care dataset and evaluate the learned policy with off-policy policy evaluation (OPPE). The results demonstrate that modelling as POMDPs yields better performance than MDPs, and that incorporating heuristic search improves sample efficiency.

Citations

• The Optimal Control of Partially Observable Markov Processes over a Finite Horizon
• Variational inference for Monte Carlo objectives
• Safe and Efficient Off-Policy Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• On Improving Deep Reinforcement Learning for POMDPs
• Deep Variational Reinforcement Learning for POMDPs

Cited by

• Exploring a Graph-based Approach to Offline Reinforcement Learning for Sepsis Treatment
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare

Related

• The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making
• Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation
• POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning
• Learning Cost-Effective Treatment Regimes using Markov Decision Processes
• Learning to search efficiently for causally near-optimal treatments