Apprenticeship learning (AL) aims to induce decision-making policies by observing and imitating expert demonstrations.Existing AL approaches typically rely on online interactions and assume that the demonstrations follow a single reward function.Nevertheless, in real-world human-centric applications, policies are usually learned in an offline setting, with the demonstrations driven by multiple reward functions that evolve over time.To address these challenges, we introduce a novel AL framework: Time-aware Hierarchical EM Energy-based Sub-trajectory (THEMES) clustering.We evaluate the effectiveness of THEMES in two challenging human-centric domains -healthcare and education.Our experimental results across multiple datasets demonstrate that THEMES can accurately induce policies, outperforming competitive baselines and ablations, demonstrating its potential for tackling a broad range of complex, real-world human-centric tasks.

Rewards are the critical signals for Reinforcement Learning (RL) algorithms to learn the desired behavior in a sequential multi-step learning task. However, when these rewards are delayed and noisy in nature, the learning process becomes more challenging. The temporal Credit Assignment Problem (CAP) is a well-known and challenging task in AI. While RL, especially Deep RL, often works well with immediate rewards but may fail when rewards are delayed or noisy, or both. In this work, we propose delegating the CAP to a Neural Network-based algorithm named InferNet that explicitly learns to infer the immediate rewards from the delayed and noisy rewards. The effectiveness of InferNet was evaluated on three online RL tasks: a GridWorld, a CartPole, and 40 Atari games; and two offline RL tasks: GridWorld and a real-life Sepsis treatment task. The effectiveness of InferNet rewards is compared to that of immediate and delayed rewards in two settings: with and without noise. For the offline RL tasks, it is also compared to a strong baseline, InferGP [7]. Overall, our results show that InferNet is robust to delayed or noisy reward functions, and it could be used effectively for solving the temporal CAP in a wide range of RL tasks, when immediate rewards are not available or they are noisy.

Healthcare systems are struggling with increasing workloads that adversely affect quality of care and patient outcomes. When clinical practitioners have to make countless medical decisions, they may not always able to make them consistently or spend time on them. In this work, we formulate clinical decision making as a reinforcement learning (RL) problem and propose a human-controlled machine-assisted (HC-MA) decision making framework whereby we can simultaneously give clinical practitioners (the humans) control over the decision-making process while supporting effective decision-making. In our HC-MA framework, the role of the RL agent is to nudge clinicians only if they make suboptimal decisions at critical moments. This framework is supported by a general Critical Deep RL (Critical-DRL) approach, which uses Long-Short Term Rewards (LSTRs) and Critical Deep Q-learning Networks (CriQNs). Critical-DRL’s effectiveness has been evaluated in both a GridWorld game and real-world datasets from two medical systems: a large health system in the northeast of USA, referred as NEMed and Mayo Clinic in Rochester, Minnesota, USA for septic patient treatment. We found that our Critical-DRL approach, by which decisions are made at critical junctures, is as effective as a fully executed DRL policy and moreover, it enables us to identify the critical moments in the septic treatment process, thus greatly reducing burden on medical decision-makers by allowing them to make critical clinical decisions without negatively impacting outcomes.

Sepsis is a life-threatening organ dysfunction and a disease of astronomical burden. Septic shock, the most severe complication of sepsis, leads to a mortality rate as high as 50%. However, septic shock prevention is extremely challenging because individual patients often have very different disease progression, and thus the timings of medical interventions can play a key role in their effectiveness. Recently, reinforcement learning (RL) methods like deep Q-learning networks (DQN) have shown great promise in developing effective treatments for preventing septic shock. In this work, we propose MTA-TQN, a Multi-view -Temporal Abstraction mechanism within a Time-aware deep Q-learning Network framework for this task. More specifically, 1) MTA-TQN leverages irregular time intervals to discount expected return which would prevent systemic overestimations caused by temporal discount errors; 2) it learns both short and long-range dependencies with multi-view temporal abstractions which would reduce bias to a specific series of observations for a single state. The effectiveness of MTA-TQN is validated on two hard exploration Atari games and the septic shock prevention task using real-world EHRs. Our results demonstrate that both time-awareness and multi-view temporal abstraction are essential to induce effective policies, particularly with irregular time-series data. In the septic shock prevention task, while the top 10% of patients whose treatments agreed with DQN induced policy experienced a 17% septic shock rate, our MTA-TQN policies achieved a 5.7% septic shock rate.

Modeling patient disease progression using Electronic Health Records (EHRs) is critical to assist clinical decision making. While most of prior work has mainly focused on developing effective disease progression models using EHRs collected from an individual medical system, relatively little work has investigated building robust yet generalizable diagnosis models across different systems. In this work, we propose a general domain adaptation (DA) framework that tackles two categories of discrepancies in EHRs collected from different medical systems: one is caused by heterogeneous patient populations (covariate shift) and the other is caused by variations in data collection procedures (systematic bias). Prior research in DA has mainly focused on addressing covariate shift but not systematic bias. In this work, we propose an adversarial domain separation framework that addresses both categories of discrepancies by maintaining one globally-shared invariant latent representation across all systems through an adversarial learning process, while also allocating a domain-specific model for each system to extract local latent representations that cannot and should not be unified across systems. Moreover, our proposed framework is based on variational recurrent neural network (VRNN) because of its ability to capture complex temporal dependencies and handling missing values in time-series data. We evaluate our framework for early diagnosis of an extremely challenging condition, septic shock, using two real-world EHRs from distinct medical systems in the U.S. The results show that by separating globally-shared from domain-specific representations, our framework significantly improves septic shock early prediction performance in both EHRs and outperforms the current state-of-the-art DA models.

Modeling patient disease progression using Electronic Health Records (EHRs) is crucial to assist clinical decision making. In recent years, deep learning models such as Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) have shown great success in handling sequential multivariate data, such as EHRs. Despite their great success, it is often difficult to interpret and visualize patient disease progression learned from these models in a meaningful yet unified way. In this work, we present MuLan: a Multilevel Language-based representation learning framework that can automatically learn a hierarchical representation for EHRs at entry, event, and visit levels. We validate MuLan on modeling the progression of an extremely challenging disease, septic shock, by using real-world EHRs. Our results showed that these unified multilevel representations can be utilized not only for interpreting and visualizing the latent mechanism of patients' septic shock progressions but also for early detection of septic shock.

Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) have become the state-of-the-art approaches for mining Electronic Health Records (EHRs). Generally speaking, RNN extracts the temporal dependency among features as a time series of hidden states, whereas CNN summarizes the local patterns among features as a set of feature maps. Many studies have leveraged their complementary effects by using a stack of neural network layers with CNN on the input, followed by RNN layers for the output. However, the feature representations learned by these two types of neural networks are often hard to be visualized and interpreted in a unified way. In this work, we propose a general framework which represents the extracted temporal relationships and local patterns in a unified and systematic way through facial representations that have evolving emotional expressions based on a patient's health conditions. This form of feature representation not only improves the potential to visualize EHRs, but also further benefits our downstream task on early prediction of septic shock. More specifically, we show that our proposed framework consistently out-performed all other baseline models including various deep learning models for sepsis shock early prediction.

Since Density Peak Clustering (DPC) algorithm was proposed in 2014, it has drawn lots of interest in various domains. As a clustering method, DPC features superior generality, robustness, flexibility and simplicity. There are however two main roadblocks for its practical adoptions, both centered around the selection of cutoff distance, the single critical hyperparameter of DPC. This work proposes an improved algorithm named Streamlined Density Peak Clustering (SDPC). SDPC speeds up DPC executions on a sequence of cutoff distances by 2.2-8.8X while at the same time reducing memory usage by a magnitude. As an algorithm preserving the original semantic of DPC, SDPC offers an efficient and scalable drop-in replacement of DPC for data clustering.

Deep neural network models, especially Long Short Term Memory (LSTM), have shown great success in analyzing Electronic Health Records (EHRs) due to their ability to capture temporal dependencies in time series data. When applying the deep learning models to EHRs, we are generally confronted with two major challenges: high rate of missingness and time irregularity. Motivated by the original PACIFIER framework which utilized matrix decomposition for data imputation, we applied and further extended it by including three components: forecasting future events, a time-aware mechanism, and a subgroup basis approach. We evaluated the proposed framework with real-world EHRs which consists of 52,919 visits and 4,224,567 events on a task of early prediction of septic shock. We compared our work against multiple baselines including the original PACIFIER using both LSTM and Time-aware LSTM (T-LSTM). Experimental results showed that our proposed framework significantly outperformed all competitive baseline approaches. More importantly, the extracted interpretative latent patterns from subgroups could shed some lights for clinicians to discover the progression of septic shock patients.

Sepsis is a leading cause of death over the world and septic shock, the most severe complication of sepsis, reaches a mortality rate as high as 50%. Early diagnosis and treatment can prevent most morbidity and mortality. Nowadays, the increasing availability of the electronic health records (EHRs) has generated great interests in developing models to predict acute medical conditions such as septic shock. However, septic shock prediction faces two major challenges : 1) how to capture the informative progression of septic shock in a long visit to hospital of a patient; and 2) how to obtain reliable predictions without well-established moment-by-moment ground-truth labels for septic shock. In this work, we proposed a generic framework to predict septic shock based on Long-Short Term Memory (LSTM) model, which is capable of memorizing temporal dependencies over a long period. The framework integrates two levels of imperfect yet informative labels to jointly learn the discriminative patterns of septic shock: ICD-9 code as the visit-level label and the clinical criteria designed by domain experts as the moment-by-moment event-level label. We evaluate our method on a real-world data extracted from an EHR system constituted by 12,954 visits and 1,348,625 events, and compare it against multiple baselines. The robustness of the method is validated using three sets of clinician-proposed adjusted ground-truth labels. Also, we explore whether the framework is effective for the early prediction of the patients developing septic shock. The experimental results demonstrate the superiority of our proposed method in the task of septic shock prediction.