Trajectory Inspection: A Method for Iterative Clinician-Driven Design of\n Reinforcement Learning Studies

2020/10/08 by Christina X. Ji, Michael Oberst, Ji, Christina X. +5 · 3 citations
Computer Science · Health Professions · Medicine · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Healthcare Operations and Scheduling Optimization #Hemodynamic Monitoring and Therapy #Machine Learning (cs.LG) #Machine Learning in Healthcare #Sepsis Diagnosis and Treatment

paper · pdf · doi:10.48550/arxiv.2010.04279

openalex publication_date 2020/10/08 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) has the potential to significantly improve\nclinical decision making. However, treatment policies learned via RL from\nobservational data are sensitive to subtle choices in study design. We\nhighlight a simple approach, trajectory inspection, to bring clinicians into an\niterative design process for model-based RL studies. We identify where the\nmodel recommends unexpectedly aggressive treatments or expects surprisingly\npositive outcomes from its recommendations. Then, we examine clinical\ntrajectories simulated with the learned model and policy alongside the actual\nhospital course. Applying this approach to recent work on RL for sepsis\nmanagement, we uncover a model bias towards discharge, a preference for high\nvasopressor doses that may be linked to small sample sizes, and clinically\nimplausible expectations of discharge without weaning off vasopressors. We hope\nthat iterations of detecting and addressing the issues unearthed by our method\nwill result in RL policies that inspire more confidence in deployment.\n

Citations

• Evaluating Reinforcement Learning Algorithms in Observational Health Settings

Cited by

• Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment
• medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Support
• Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies

Related

• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation
• How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models
• Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis