To derive expressive representations from high-dimensional and sparse multi-omics samples, there has been existing research attempting to incorporate Gene Regulatory Network (GRN) as a prior knowledge to enhance deep learning models. However, these methods generally just considered homogeneous GRNs with simple structures containing just a single gene/interaction type for single omics data analysis. In this paper, we propose a new framework MoHeG that infuses the abundant knowledge in heterogeneous GRN for representation learning of multi-omics data. Particularly, MoHeG first adopts an interaction-specific graph attention network to represent the graph structure of heterogeneous GRN. Then, a self-supervised learning strategy, which combines an auto-encoder task and a contrastive learning task, is utilized to pre-train the network with unlabeled gene data in both detail and distinctiveness level. Through a fine-tuning approach, we can derive expressive representations for various downstream tasks. Our experiments demonstrate the effectiveness of the new framework on both sufficient and insufficient datasets, compared to a series of state-of-the-art baselines. MoHeG has the potential to become an advanced encoder in analysis pipeline of multi-omics data.

Electrocardiogram (ECG) is one of the critical diagnostic tools in healthcare. Various deep learning models, except Transformers, have been explored and applied to map ECG patterns to heart abnormalities. Transformer models have been adopted from natural language processing to computer vision with advanced features. Most recently, vision transformers show exceptional performances, even on moderate-scale datasets. However, naively applying vision transformers on electrocardiogram datasets leads to poor results. In this paper, we propose a novel network called Beat-aligned Transformer (BaT), a hierarchical Transformer that sufficiently exploits the cyclicity of ECG. We organize and treat an input ECG as multiple aligned beats instead of a single time series. In the BaT, shifted-window-based Transformer blocks (SW Block) are adopted to learn the representation for each beat, and aggregation blocks are designed to exchange information among the beat representations. Nested SW Blocks and aggregation blocks form a beat-aware hierarchical structure of BaT. In this way, the new data format and the BaT hierarchical structure boost Transformer performance on ECG classification. From the experiments on public ECG datasets, we observe BaT outperforms other Transformer-based models and achieves competitive performance compared with other state-of-the-art methods.

In recent years, with the better availability of medical data such as Electronic Health Records (EHR), more and more data mining models have been developed to explore the data-driven insights for better human health. However, there are many challenges for analyzing EHR such as high-dimensionality, temporality, sparsity, etc., which make the data-driven models less reliable. Medical knowledge graph (MKG), which encodes comprehensive knowledge about the medical concepts and relationships extracted from medical literature, holds great promise to regularize the data-driven models as prior knowledge. Nonetheless, the MKGs are typically not complete, which limits its utility in helping with the data mining process. In this paper, we propose a mutual enhancement framework MendMKG for predictive modeling of clinical events with both EHR and MKG. In particular, MendMKG first conducts a self-supervised learning strategy to simultaneously pre-train a graph attention network for embedding nodes and complete the MKG. It iteratively performs (1) an embedding-based knowledge graph completion module to derive missing edges, (2) and a reconstruction module of unlabeled EHR data to select high-quality ones from these edges, which would be further appended to the MKG to update the embedding model. Through the iterations, the two modules mutually benefit each other. Then, MendMKG uses the pre-trained graph attention network and the updated MKG to generate the visit embeddings to represent patient’s historical visits, and predict the diagnosis in future visit, through a fine-tuning approach. Experimental results on real world EHR corpus are provided to demonstrate the superiority of the proposed framework, compared to a series of state-of-the-art baselines.11The source code and knowledge graph data have been anonymously uploaded to https://github.com/1317375434/MendMKG.

When taking care of complex patients with multiple morbidities, accurately predicting the occurrence of each cause-specific event is critical for designing optimal treatment plans. However, standard survival analysis cannot deal with the multiple (usually competing) adverse events and views those competing events as censored. This will result in biased estimation of the incidence rate. In this paper, we propose a deep learning based survival analysis algorithm called DeepComp to jointly predict the progress of the competing events, which can thus inform the doctors which event is more likely to hit the patient first. DeepComp constructs a multi-task recurrent neural network (RNN) and views the conditional probability of each competing event at each time point as the output of each RNN cell. Then the probability chain rule is utilized to combine them together. In this way, the survival probability and the risk for each competing event over the time space are obtained. The multitask structure not only prevents the model from unreasonable censoring but also aids the model in capturing the complex hidden association among the competing events. A novel penalty is added to the loss function to better discriminate the competing risks for each particular patient, which could benefit treatment decision-making. We conduct comprehensive experiments on two real-world clinical data sets and one synthetic data set. The proposed DeepComp method achieves significant performance improvement compared to the state-of-the-art baseline methods.

Representation learning (RL) plays an important role in extracting proper representations from complex medical data for various analyzing tasks, such as patient grouping, clinical endpoint prediction and medication recommendation. Medical data can be divided into two typical categories, outpatient and inpatient, that have different data characteristics. However, few existing RL methods are specially designed for inpatients data, which have strong temporal relations and consistent diagnosis. In addition, for unordered medical activity set, existing medical RL methods utilize a simple pooling strategy, which would result in indistinguishable contributions among the activities for learning. In this work, we propose Inpatient2Vec, a novel model for learning three kinds of representations for inpatient, including medical activity, hospital day and diagnosis. A multilayer self-attention mechanism with two training tasks is designed to capture the inpatient data characteristics and process the unordered set. Using a real-world dataset, we demonstrate that the proposed approach outperforms the competitive baselines on semantic similarity measurement and clinical events prediction tasks.

Patient Electronic Health Records (EHR) data consist of sequences of patient visits over time. Sequential prediction of patients' future clinical events (e.g., diagnoses) from their historical EHR data is a core research task and motives a series of predictive models including deep learning. The existing research mainly adopts a classification framework, which treats the observed and unobserved events as positive and negative classes. However, this may not be true in real clinical setting considering the high rate of missed diagnoses and human errors. In this paper, we propose to formulate the clinical event prediction problem as an events recommendation problem. An end-to-end pairwise-ranking based collaborative recurrent neural networks (PacRNN) is proposed to solve it, which firstly embeds patient clinical contexts with attention RNN, then uses Bayesian Personalized Ranking (BPR) regularized by disease co-occurrence to rank probabilities of patient-specific diseases, as well as use point process to provide simultaneous prediction of the occurring time of these diagnoses. Experimental results on two real world EHR datasets demonstrate the robust performance, interpretability, and efficacy of PacRNN.

With the better availability of healthcare data, such as Electronic Health Records (EHR), more and more data analytics methodologies are developed aiming at digging insights from them to improve the quality of care delivery. There are many challenges on analyzing EHR, such as high dimensionality and event sparsity. Moreover, different from other application domains, the EHR analysis algorithms need to be highly interpretable to make them clinically useful. This makes representation learning from EHRs of key importance. In this paper, we propose an algorithm called Predictive Task Guided Tensor Decomposition (TaGiTeD), to analyze EHRs. Specifically, TaGiTeD learns event interaction patterns that are highly predictive for certain tasks from EHRs with supervised tensor decomposition. Compared with unsupervised methods, TaGiTeD can learn effective EHR representations in a more focused way. This is crucial because most of the medical problems have very limited patient samples, which are not enough for unsupervised algorithms to learn meaningful representations form. We apply TaGiTeD on real world EHR data warehouse and demonstrate that TaGiTeD can learn representations that are both interpretable and predictive.

The semantic description of remote sensing image is a useful and meaningful task, which can help us to get a better understanding of the scene depicted in the remote sensing images and make better use of the remote sensing images. Nature language provides good solution for describing the semantic information of remote sensing images. Nature language description of a remote sensing image is to generate a meaningful sentence given a remote sensing image. This paper presents a novel method based on deep learning. First, a convolutional neural network is utilized to detect the main objects of the remote sensing images. Then a recurrent neural network language model is utilized to generate the natural language descriptions of the objects which are detected in the first step. Experimental results on a set of remote sensing images demonstrate that the proposed method is able to generate desirable description of the scene.

Anomaly detection in healthcare data like patient records is no trivial task. The anomalies in these datasets are often caused by mismatches between different types of feature, e.g., medications that do not match with the diagnoses. Existing anomaly detection methods do not perform well when detecting "mismatches" between multiple types of feature, especially when the feature space is high-dimensional and sparse. This paper introduces a novel anomaly detection paradigm: Probabilistic-Mismatch Anomaly Detection (PMAD), which detects mismatches between features by modeling a normal instance with a common latent probability distribution that governs the generation of all types of feature. Under this paradigm, the target of anomaly detection is to find instances with dissimilar latent distributions. We further propose Topical PMAD based on an extended Latent Dirichlet Allocation (LDA) model, which is able to capture the latent relationship between features in a high-dimensional space. Experiments on both synthetic data and real-world patient records show that Topical PMAD can effectively detect anomalies with mismatched features, and is highly robust against high-dimensional data as well as inaccurate model selection. The real-world anomalies detected on a patient record dataset show a promising application prospect.

A Care Pathway is a knowledge-centric process to guide clinicians to provide evidence-based care to patients with specific conditions. One existing problem for care pathways is that they often fail to reflect the best clinical practice as a result of not being adequately updated. A better understanding of the gaps between a care pathway and real practice requires aligning patient records with the pathway. Patient records are unlabeled in practice making it difficult to align them with a care pathway which is inherently complex due to its representation as a hierarchical and declarative process model (HDPM). This paper proposes to solve this problem by developing a Hierarchical Markov Random Field (HMRF) method so that a set of patient records can best fit a given care pathway. We validate the effectiveness of the method with experiments on both synthesized data and real clinical data.

We present a novel performance-driven approach to animating cartoon faces starting from pure 2D drawings. A 3D approximate facial model automatically built from front and side view master frames of character drawings is introduced to enable the animated cartoon faces to be viewed from angles different from that in the input video. The expressive mappings are built by artificial neural network (ANN) trained from the examples of the real face in the video and the cartoon facial drawings in the facial expression graph for a specific character. The learned mapping model makes the resultant facial animation to properly get the desired expressiveness, instead of a mere reproduction of the facial actions in the input video sequence. Furthermore, the lit sphere, capturing the lighting in the painting artwork of faces, is utilized to color the cartoon faces in terms of the 3D approximate facial model, reinforcing the hand-drawn appearance of the resulting facial animation. We made a series of comparative experiments to test the effectiveness of our method by recreating the facial expression in the commercial animation. The comparison results clearly demonstrate the superiority of our method not only in generating high quality cartoon-style facial expressions, but also in speeding up the animation production of cartoon faces. Copyright © 2011 John Wiley & Sons, Ltd.

Abstract In this paper, we present a novel framework for creating cartoon facial animation from multi‐view hand‐drawn sketches. The input sketches are first employed to construct a base mesh model by using a hybrid sketch‐based method. The model is then deformed for each key viewpoint, yielding a set of models that closely match the corresponding sketches. We introduce a view‐dependent facial expression space defined by the key viewpoints and the basic emotions to generate various facial expressions viewed from arbitrary angles. The output facial animation conforms to the input sketches and maintains frame‐to‐frame correspondence. We demonstrate the potential of our approach through an easy‐to‐use system, where the animating of cartoon faces is automated once the user accomplishes sketching and configuration. Copyright © 2010 John Wiley & Sons, Ltd.