Transformers based on Self-Attention (SA) mechanism have demonstrated unrivaled superiority in numerous areas. Compared to RNN-based networks, Transformers can learn the temporal dependency representation of an entire sequence in parallel, while efficiently dealing with long-range dependencies. However, the $\mathcal {O}(L^{2})$O(L2) ($L$L denotes the length of the sequence) computational complexity of the SA mechanism and the high memory usage make the construction cost of the Transformer-based model prohibitively expensive. To address these challenges, we propose a Transformer-like model, HPformer: Low-Parameter Transformer with Temporal Dependency Hierarchical Propagation. HPformer first chunks the sequence into $K$K ($K = \left\lceil \log {L} \right\rceil + 1$K=logL+1, $\left\lceil \cdot \right\rceil$· denotes ceiling operation) sequence segments, then leverages the hierarchical propagation mechanism with $\mathcal {O}(L)$O(L) computational complexity to learn the temporal dependencies between the segments and within the segments, and ultimately generates $K$K vectors as $Key$Key matrices. This reduces the complexity of the SA mechanism from $\mathcal {O}(L^{2})$O(L2) to $\mathcal {O}(L\log {L})$O(LlogL). In addition, we employ a strategy of sharing $Key$Key and $Value$Value matrices between layers to build the HPformer, thus reducing memory usage. Extensive experiments based on public health informatics benchmark and Long-Range Arena (LRA) benchmark have demonstrated that HPformer has advantages over Transformer-based models in terms of memory usage and efficiency.

Accurate prediction of patients’ ICU transfer events is of great significance for improving ICU treatment efficiency. ICU transition prediction task based on Electronic Health Records (EHR) is a temporal mining task like many other health informatics mining tasks. In the EHR-based temporal mining task, existing approaches are usually unable to mine and exploit patterns used to improve model performance. This article proposes a network based on Interval Pattern-Aware, Multi-Scale Interval Pattern-Aware (MSIPA) network. MSIPA mines different interval patterns in temporal EHR data according to the short, medium, and long intervals. MSIPA utilizes the Scaled Dot-Product Attention mechanism to query the contexts corresponding to the three scale patterns. Furthermore, Transformer will use all three types of contextual information simultaneously for ICU transfer prediction. Extensive experiments on real-world data demonstrate that an MSIPA network outperforms state-of-the-art methods.

Mining Electronic Health Records (EHRs) is of great significance to improve the efficiency and quality of medical services. In recent years, researchers have used deep recurrent neural networks to predict patients' next-period prescriptions. The main challenges of predicting next-period prescriptions are as follows: i) The latent interdependence information which can be utilized to enhance the representation of the data between heterogeneous features is dynamic. However, most existing approaches do not consider capturing and utilizing this information. ii) Conventional recurrent neural networks cannot store historical interdependent information between heterogeneous features and take advantage of it. To address these challenges, We propose a novel attention mechanism named Cross Attention (CA) that can capture the interdependence between two sequences. We further propose three types of recurrent neural networks that can capture and utilize current and historical latent interdependence between the two sequences. Extensive experiments on real-world data demonstrate that DSCA networks can capture the interdependence information between sequences and outperform state-of-the-art methods.