The advent of multimodal large language models (MLLMs) has sparked interest in their application to electrocardiogram (ECG) analysis. However, existing ECG-focused MLLMs primarily focus on report generation tasks, often limited to single 12-lead, short-duration (10s) ECG inputs, thereby underutilizing the potential of MLLMs. To this end, we aim to develop a MLLM for ECG analysis that supports a broader range of tasks and more flexible ECG inputs. However, existing ECG-QA datasets are often monotonous. To address this gap, we first constructed the anyECG dataset, which encompasses a wide variety of tasks, including report generation, abnormal waveform localization, and open-ended question answering. In addition to standard hospital ECGs, we introduced long-duration reduced-lead ECGs for home environments and multiple ECG comparison scenarios commonly encountered in clinical practice. Furthermore, we propose the anyECG-chat model, which supports dynamic-length ECG inputs and multiple ECG inputs. We trained the model using a three-stage curriculum training recipe with the anyECG dataset. A comprehensive evaluation was conducted, demonstrating that anyECG-chat is capable of supporting various practical application scenarios, including not only common report generation tasks but also abnormal waveform localization for long-duration reduced-lead ECGs in home environments and comprehensive comparative analysis of multiple ECGs.

Non-communicable disease is the leading cause of death, emphasizing the need for accurate prediction of disease progression and informed clinical decision-making. Machine learning (ML) models have shown promise in this domain by capturing non-linear patterns within patient features. However, existing ML-based models cannot provide causal interpretable predictions and estimate treatment effects, limiting their decision-making perspective. In this study, we propose a novel model called causal trajectory prediction (CTP) to tackle the limitation. The CTP model combines trajectory prediction and causal discovery to enable accurate prediction of disease progression trajectories and uncover causal relationships between features. By incorporating a causal graph into the prediction process, CTP ensures that ancestor features are not influenced by the treatment of descendant features, thereby enhancing the interpretability of the prediction. By estimating the bounds of treatment effects, the CTP provides valuable insights for clinical decision-making. We evaluate the performance of the CTP using simulated and real medical datasets. Experimental results demonstrate that our model has the potential to assist clinical decisions.

Visual information is often used as a complementary cue for automatic speech recognition in noisy environments. Most previous studies utilize visual information of target speakers (e.g., lip movements) to improve the recognition performance of audio-visual speech recognition (AVSR) models. However, it remains unclear whether visual information of background sound can benefit automatic speech recognition. Our study proceeds in this regard by constructing a new audiovisual dataset and devising an AVSR model. The new dataset, Audio-Visual Natural Scenes (abbreviated as AVNS) dataset, consists of 11 types of natural scenes (around 31.3 hours) and was recorded through professional recording devices. The AVNS dataset provides audio and visual signals of common background noises in natural acoustic scenes. The AVSR model was designed based on a representation learning framework called AV-HuBERT, which could fuse representations of audio and visual modalities for automatic speech recognition. In this work, we combined the AVNS dataset (providing background sound) with the largest benchmark LRS3 dataset (providing target speech) to create adverse noise conditions for the AVSR model. The results showed that incorporating visual information synchronized with background noises greatly improved model performance (reducing WER by up to 4.9%) in noisy environments. These findings demonstrate that noise-related visual information can contribute to model performance in automatic speech recognition.

Objective: To combine medical knowledge and medical data to interpretably predict the risk of disease. Methods: We formulated the disease progression prediction task as a random walk along a knowledge graph (KG). Specifically, we build a KG to record relationships between diseases and risk factors according to validated medical knowledge. Then, an object walks along the KG. It starts walking at a patient entity, which connects the KG based on the patient’s current diseases or risk factors and stops at a disease entity representing the predicted disease. The trajectory generated by the object represents an interpretable disease progression path of the given patient. The dynamics of the object are controlled by a policy-based reinforcement learning module, which is trained by electronic health records (EHRs). Experiments: We utilized three real-world EHR datasets to evaluate the performance of our model. In the disease progression prediction task, our model achieves 0.743, 0.639, and 0.643 in terms of macro area under the curve (AUC) in predicting 53 circulation system diseases in the three datasets, respectively. This performance is comparable to medical research’s commonly used machine learning models. In qualitative analysis, our clinical collaborator reviewed the disease progression paths generated by our model and advocated their interpretability and reliability. Conclusion: Experimental results validate the proposed model in interpretably evaluating and optimizing disease progression prediction. Significance: Our work contributes to leveraging the potential of medical knowledge and medical data jointly for interpretable prediction tasks.

Dialogue state tracking (DST) is a component of the task oriented dialogue system. It is responsible for extracting and managing slots, where each slot represents a part of the information to accomplish a task, and slot value is updated recurrently in each dialogue turn. However, many DST models cannot update slot values appropriately. These models may repeatedly inherit wrong slot values extracted in previous turns, resulting in the fail of the entire DST task. They cannot update indirectly mentioned slots well, either. This study designed a model with a mentioned slot pool (MSP) to tackle the update problem. The MSP is a slot specific memory that records all mentioned slot values that may be inherited, and our model updates slot values according to the MSP and the dialogue context. Our model rejects inheriting the previous slot value when it predicates the value is wrong. Then, it extracts the slot value from the current dialogue context. As the contextual information accumulates, the new value is more likely to be correct. It also can track the indirectly mentioned slot by picking a value from the MSP. Experimental results showed our model reached state of the art DST performance on MultiWOZ datasets.

Massively available longitudinal data about long-term disease trajectories of patients provides a golden mine for the understanding of disease progression and efficient health service delivery. It calls for quantitative modeling of disease progression, which is a tricky problem due to the complexity of the disease progression process as well as the irregularity of time documented in trajectories. In this study, we tackle the problem with the goal of predictively analyzing disease progression. Specifically, we propose a novel Variational Hawkes Process (VHP) model to generalize disease progression and predict future patient states based on the clinical observational data of past disease trajectories. First, Hawkes Process captures the intensity of irregular visits in a trajectory documented to medical facilities and controls the aforementioned information flowing into future visits. Thereafter, the captured intensity is incorporated into a Variational Auto-Encoder to generate the representation of the future partial disease trajectory for a target patient in a predictive manner. To further improve the prediction performance, we equip the proposed model with a disease trajectory discriminator to distinguish the generated trajectories from real ones. We evaluate the proposed model on two public datasets from the MIMIC-III database pertaining to heart failure and sepsis patients, respectively, and one real-world dataset from a Chinese hospital pertaining to heart failure patients with multiple admissions. Experimental results demonstrate that the proposed model significantly outperforms state-of-the-art baselines, and may derive a set of practical implications that can benefit a wide spectrum of management and applications on disease progression.

Healthcare process leaves patient treatment trajectory (PTT), described as a sequence of interdependent clinical events affiliated with a large volume of longitudinal therapy and treatment information. Predicting the future clinical event in PTT, as a vital and essential task for providing insights into the entire treatment trajectory, can serve as an efficient and proactive altering service for health service delivery. However, it is challenging because there are long-term dependencies between clinical events, which are irregularly distributed along the temporal axis with varying time intervals. This characteristic inevitably impedes the performance of clinical event prediction (CEP) using the existing approaches. To address this challenge, we propose a novel approach to learn representative and discriminative PTT features for CEP. In detail, multivariate Hawkes process (HP) is adopted to uncover the mutual excitation intensities between clinical event pairs in an interpretable manner. Thereafter, the captured spontaneous and interactional intensities of events are incorporated into recurrent neural networks (RNN) to encode PTT in latent representations, while jointly performing the CEP task based on the extracted trajectory representations. We evaluate the performance of the proposed approach on a real clinical dataset consisting of 13,545 visits of 2,102 heart failure patients. Compared to state-of-the-art methods, our best model achieves 6.4% and 4.1% AUC performance gains on three-months and one-year CEP tasks, respectively. The experimental results demonstrate that the proposed approach outperforms state-of-the-art models in CEP, and can be profitably exploited as a basis for PTT analysis and optimization.