Offline reinforcement learning with uncertainty for treatment strategies in sepsis

2021/07/09 by Ran Liu, Liu, Ran, Joseph L. Greenstein +9 · 5 citations
Computer Science · Medicine · #Adverse effect #Artificial Intelligence (cs.AI) #Clinical Reasoning and Diagnostic Skills #Computer science #FOS: Computer and information sciences #I.2.1 #Intensive care medicine #Internal medicine #Intervention (counseling) #Machine Learning (cs.LG) #Machine Learning in Healthcare #Machine learning #Medicine #Personalization #Psychological intervention #Reinforcement learning #Sepsis #Sepsis Diagnosis and Treatment #Septic shock #Surgery #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2107.04491

published in arXiv (Cornell University) (Cornell University) · 25 pages, 8 figures

openalex publication_date 2021/07/09 · arxiv created 2021/09/22 · arxiv updated 2021/09/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Guideline-based treatment for sepsis and septic shock is difficult because sepsis is a disparate range of life-threatening organ dysfunctions whose pathophysiology is not fully understood. Early intervention in sepsis is crucial for patient outcome, yet those interventions have adverse effects and are frequently overadministered. Greater personalization is necessary, as no single action is suitable for all patients. We present a novel application of reinforcement learning in which we identify optimal recommendations for sepsis treatment from data, estimate their confidence level, and identify treatment options infrequently observed in training data. Rather than a single recommendation, our method can present several treatment options. We examine learned policies and discover that reinforcement learning is biased against aggressive intervention due to the confounding relationship between mortality and level of treatment received. We mitigate this bias using subspace learning, and develop methodology that can yield more accurate learning policies across healthcare applications.

Citations

• Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock*
• Fluid Therapy in Resuscitated Sepsis
• Deep Exploration via Bootstrapped DQN
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Related

• Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
• Model-Based Reinforcement Learning for Sepsis Treatment
• Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation
• Deep Reinforcement Learning for Sepsis Treatment
• How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models