Healthcare systems are struggling with increasing workloads that adversely affect quality of care and patient outcomes. When clinical practitioners have to make countless medical decisions, they may not always able to make them consistently or spend time on them. In this work, we formulate clinical decision making as a reinforcement learning (RL) problem and propose a human-controlled machine-assisted (HC-MA) decision making framework whereby we can simultaneously give clinical practitioners (the humans) control over the decision-making process while supporting effective decision-making. In our HC-MA framework, the role of the RL agent is to nudge clinicians only if they make suboptimal decisions at critical moments. This framework is supported by a general Critical Deep RL (Critical-DRL) approach, which uses Long-Short Term Rewards (LSTRs) and Critical Deep Q-learning Networks (CriQNs). Critical-DRL’s effectiveness has been evaluated in both a GridWorld game and real-world datasets from two medical systems: a large health system in the northeast of USA, referred as NEMed and Mayo Clinic in Rochester, Minnesota, USA for septic patient treatment. We found that our Critical-DRL approach, by which decisions are made at critical junctures, is as effective as a fully executed DRL policy and moreover, it enables us to identify the critical moments in the septic treatment process, thus greatly reducing burden on medical decision-makers by allowing them to make critical clinical decisions without negatively impacting outcomes.

Sepsis is a life-threatening organ dysfunction and a disease of astronomical burden. Septic shock, the most severe complication of sepsis, leads to a mortality rate as high as 50%. However, septic shock prevention is extremely challenging because individual patients often have very different disease progression, and thus the timings of medical interventions can play a key role in their effectiveness. Recently, reinforcement learning (RL) methods like deep Q-learning networks (DQN) have shown great promise in developing effective treatments for preventing septic shock. In this work, we propose MTA-TQN, a Multi-view -Temporal Abstraction mechanism within a Time-aware deep Q-learning Network framework for this task. More specifically, 1) MTA-TQN leverages irregular time intervals to discount expected return which would prevent systemic overestimations caused by temporal discount errors; 2) it learns both short and long-range dependencies with multi-view temporal abstractions which would reduce bias to a specific series of observations for a single state. The effectiveness of MTA-TQN is validated on two hard exploration Atari games and the septic shock prevention task using real-world EHRs. Our results demonstrate that both time-awareness and multi-view temporal abstraction are essential to induce effective policies, particularly with irregular time-series data. In the septic shock prevention task, while the top 10% of patients whose treatments agreed with DQN induced policy experienced a 17% septic shock rate, our MTA-TQN policies achieved a 5.7% septic shock rate.