Transformers based on Self-Attention (SA) mechanism have demonstrated unrivaled superiority in numerous areas. Compared to RNN-based networks, Transformers can learn the temporal dependency representation of an entire sequence in parallel, while efficiently dealing with long-range dependencies. However, the $\mathcal {O}(L^{2})$O(L2) ($L$L denotes the length of the sequence) computational complexity of the SA mechanism and the high memory usage make the construction cost of the Transformer-based model prohibitively expensive. To address these challenges, we propose a Transformer-like model, HPformer: Low-Parameter Transformer with Temporal Dependency Hierarchical Propagation. HPformer first chunks the sequence into $K$K ($K = \left\lceil \log {L} \right\rceil + 1$K=logL+1, $\left\lceil \cdot \right\rceil$· denotes ceiling operation) sequence segments, then leverages the hierarchical propagation mechanism with $\mathcal {O}(L)$O(L) computational complexity to learn the temporal dependencies between the segments and within the segments, and ultimately generates $K$K vectors as $Key$Key matrices. This reduces the complexity of the SA mechanism from $\mathcal {O}(L^{2})$O(L2) to $\mathcal {O}(L\log {L})$O(LlogL). In addition, we employ a strategy of sharing $Key$Key and $Value$Value matrices between layers to build the HPformer, thus reducing memory usage. Extensive experiments based on public health informatics benchmark and Long-Range Arena (LRA) benchmark have demonstrated that HPformer has advantages over Transformer-based models in terms of memory usage and efficiency.

At present, the microservice architecture is widely used in modern software development for its flexibility and scalability. However, the huge data scale and complex invocation relationships between services make the root cause localization of faults in microservice systems extremely difficult. Some of the current root cause localization methods based on metrics data use history fault data to effectively improve the localization results, but there are challenges such as not representing history faults effectively and limiting the localization results to repetitive faults that have occurred in history. In this paper, we propose an automatic root cause localization framework MicroHFRCL based on the history fault library to address the above issues. MicroHFRCL constructs an instance causal graph based on metric data for causal analysis. The instance causal graph is weighted by encoding the anomalous subgraph and calculating the similarity of history faults. The PageRank algorithm is used to locate the root cause of faults. Among them, MicroHFRCL learns the structure and feature information of fault anomalous subgraphs through GCN and Transformer models, achieving effective representation of history faults and fast calculation of similarity in history fault codes, improving the efficiency of repetitive fault localization, and effectively solving the problem of the limitation of using history faults for root cause localization results. We implemented MicroHFRCL and tested it on the fault dataset collected by a benchmark microservice test system. Compared with the latest baseline models, MicroHFRCL has significantly improved the localization accuracy, and can also achieve good results in the case of small-scale history faults.

Approaches predicting the results of mutation testing by machine learning have been proposed to reduce the cost of mutation testing. The predictive approaches based on PIE theory and approaches based on natural language have been proposed. However, both PIE-based and natural language-based approaches have disadvantages, leading to a reduction in effectiveness at the test case level prediction. In order to predict at the test case level and improve the effectiveness of prediction, we propose Natural Language and PIE Predictive Mutation Testing (NLPIE-PMT), which combines advantages of PIE-based and natural language-based approaches and predict whether each test case kills each mutant in the cross-version scenario. The experimental results on subjects in Defects4J show that NLPIE-PMT can predict whether each test case kill each mutant with the average F1-score of 0.811, which is 0.135 and 0.046 higher than the PIE-based baseline and the natural language-based baseline respectively. NLPIE-PMT also performs better than the baselines in predicting mutation score.

An online service system may experience various performance faults during operation. Detecting and locating these faults after they occur can significantly impact the user experience and lead to significant losses. Therefore, it is necessary to predict faults before they occur. Existing methods for fault prediction typically only predict the possibility of fault, without providing more granular predictions, such as the type of fault. This can make troubleshooting more difficult for developers. In this paper, we propose a fine-grained fault prediction method called FSFP, which not only predicts the possibility of fault but also identifies the type of fault that may occur. The method initially collects performance monitoring metrics from the runtime system, including two types: normal operation and abnormal conditions. It then utilizes cross-attention to capture the interdependencies between these two types of monitoring metrics, followed by the construction of a multi-label classification model. We evaluated FSFP by injecting faults into a benchmark microservice system. In terms of predicting the possibility of fault, FSFP achieved a precision of 0.999, a recall of 0.998, and an F1 score of 0.999. In terms of predicting the type of fault, FSFP achieved an exact match ratio of 0.955 and a Hamming loss of 0.017. In terms of predicting six specific types of faults, FSFP achieved four optimal F1 scores.

The purpose of drug recommendation is to predict the effective and safe drug combinations required for the current visit based on the historical medical data of patients. How to better mine the hidden relationship in the medical data and effectively improve the accuracy of drug recommendation are research hotspots in the medical field. This paper proposes a Collaborative Cross-attention Drug Recommendation model (CCDR) based on patient and medical relationship representations, which mines medical data from two aspects to enhance the representation ability of the model. CCDR obtains patient representation vectors by modeling the patients’ historical sequence data using Bidirectional Gated Recurrent Unit. Meanwhile, CCDR designs a medical graph structure data learning method based on relationship division to better capture the complex association relationships among diagnoses, procedures, and drugs. Finally, the representation capability of the model is enhanced by introducing a collaborative cross-attention mechanism to fuse the information obtained from both medical sequence and graph structure data. The experimental results show that the CCDR model can effectively improve the performance of drug recommendation.

Federated learning is a new type of artificial intelligence technology. During the training process, the client transmits model parameter information instead of local data to ensure their privacy and security. But it also incurs higher communication costs. This article proposes a new federated learning pruning method, FedADP, with the aim of adaptively determining pruning ratios for each layer in each client model without infringing on client privacy, and achieving more accurate pruning effects. Our method not only reduces communication costs during the training process, but also maintains accuracy similar to the original model. We conducted experimental validation using classic models and datasets, and evaluated our scheme and traditional federated learning scheme in terms of model accuracy, communication cost, and computational cost.

Text-to-SQL aims to parse natural language problems into SQL queries, which can provide a simple interface to access large databases enabling SQL novices a quicker entry into databases. As the Text-to-SQL field is intensively studied, more and more models use GNNs to encode heterogeneous graph information in this task, and how to better obtain path information between nodes in database schema heterogeneous graphs and question-database schema heterogeneous graphs will greatly affect the effectiveness of the model parsing. Our work intends to explore the problem of solving the encoding of heterogeneous graph meta-paths in the Text-to-SQL task. Previous approaches usually use multi-layer GNNs to aggregate topological structure information between nodes. However, they ignored the structural information embedded at the edges and also failed to obtain nodes that are not directly connected but can provide contextual information through meta-paths. To solve the above problem, we propose Multi-Hop Relational Graph Attention Network based Text-to-SQL Parsing Model (MHRGATSQL) for learning topological information between nodes while obtaining semantic information embedded in the edge topology. We use multi-hop attention to modify the relational graph attention network to diffuse the attention scores throughout the network, thus increasing the “receptive field” of each layer of RGAT. Experimental results on the large-scale cross-domain Text-to-SQL dataset Spider show that our model obtains an absolute improvement of 1.7% compared to the baseline and alleviates the over-smoothing problem in the deep network model.

As auxiliary collaborative signals, the entity connectivity and relation semanticity beneath knowledge graph (KG) triples can alleviate the data sparsity and cold-start issues of recommendation tasks. Thus many works consider obtaining user and item representations via information aggregation on graph-structured data within Euclidean space. However, the scale-free graphs (e.g., KGs) inherently exhibit non-Euclidean geometric topologies, such as tree-like and circle-like structures. The existing recommendation models built in a single type of embedding space do not have enough capacity to embrace various geometric patterns, consequently, resulting in suboptimal performance. To address this limitation, we propose a KG-aware recommendation model with mixed-curvature manifolds interaction learning, namely CurvRec. On the one hand, it aims to preserve various global geometric structures in KG with mixed-curvature manifold spaces as the backbone. On the other hand, we integrate Ricci curvature into graph convolutional networks (GCNs) to capture local geometric structural properties when aggregating neighbor nodes. Besides, to exploit the expressive spatial features in KG, we incorporate interaction learning to ensure the geometric message passing between curved manifolds. Specifically, we adopt curvature-aware geodesic distance metrics to maximize the mutual information between Euclidean space and non-Euclidean spaces. Through extensive experiments, we demonstrate that the proposed CurvRec outperforms state-of-the-art baselines.

Knowledge graph (KG) has been widely used in the field of recommender systems. There are some nodes in KG that guide the occurrence of interaction behaviors. We call them guided nodes. However, the current application doesn’t take into account the guided nodes in KG. We explore the utility of guided nodes in KG. It is applied in repository recommendations. In this paper, we propose an end-to-end framework, namely Guided Node Graph Convolutional Network (GNGCN), which effectively captures the connections between entities by mining the influence of related nodes. We extract samples of each entity in KG as their guided nodes and then combine the information and bias of the guided nodes when computing the representation of a given entity. The guided nodes can be extended to multiple hops. We evaluate our model on a real-world Github dataset named Github-SKG and music recommendation dataset, and the experimental results show that the method outperforms the recommendation baselines and our model is much lighter than others.

In recent years, there has been a lot of research on deep learning for survival prediction in EHR (Electronic Health Record). At present, EHR usually contains multivariate time series data with missing values. How to better predict mortality based on such data is what many studies are currently doing. Most of the mortality prediction methods based on deep learning only pay attention to missing value filling or adjusting the model structure to enhance the mortality prediction performance, but they do not combine these two aspects well. In this paper, we propose MTSSP (Multivariate Time Series for Survival Prediction), a new method that combines missing value filling and time series classification. It takes two representations of the loss pattern: the mask and the time interval, which are combined into the recurrent neural network for the interpolation of the missing value of patient characteristics. When the missing data values are interpolated, the model combines bidirectional RNN and one-dimensional CNN to jointly capture the patient's medical behavior from a global and local perspective to enhance the representation ability of data information, thereby improving the prediction accuracy of the model. In the end, we conducted our mortality prediction experiments on the real-world emergency MIMIC-III dataset and MIMIC-IV dataset. The experimental results demonstrate that the proposed approach has been shown to significantly outperform other approaches.

The explosive growth of machine learning (ML) methods is overloading users with choices for learning tasks. Method recommendation aims to alleviate this problem by selecting the most appropriate ML methods for given learning tasks. Recent research shows that the descriptive and structural information of the knowledge graphs (KGs) can significantly enhance the performance of ML method recommendation. However, existing studies have not fully explored the descriptive information in KGs, nor have they effectively exploited the descriptive and structural information to provide the necessary supervision. To address these limitations, we distinguish descriptive attributes from the traditional relationships in KGs with the rest as structural connections to expand the scope of KG descriptive information. Based on this insight, we propose the Cross-modal Knowledge Graph Contrastive learning (CKGC) approach, which regards information from descriptive attributes and structural connections as two modalities, learning informative node representations by maximizing the agreement between the descriptive view and the structural view. Through extensive experiments, we demonstrate that CKGC significantly outperforms the state-of-the-art baselines, achieving around 2% higher accurate click-through-rate (CTR) prediction, over 30% more accurate top-10 recommendation, and over 50% more accurate top-20 recommendation compared to the best performing existing approach.

Drug recommendation task based on the deep learning model has been widely studied and applied in the health care field in recent years. However, the accuracy of drug recommendation models still needs to be improved. In addition, the existing recommendation models either give only one recommendation (however, there may be a variety of drug combination options in practice) or can not provide the confidence level of the recommended result. To fill these gaps, a Drug Recommendation model based on Message Propagation neural network (denoted as DRMP) is proposed in this paper. Then, the Drug-Drug Interaction (DDI) knowledge is introduced into the proposed model to reduce the DDI rate in recommended drugs. Finally, the proposed model is extended to Bayesian Neural Network (BNN) to realize multiple recommendations and give the confidence of each recommendation result, so as to provide richer information to help doctors make decisions. Experimental results on public data sets show that the proposed model is superior to the best existing models.

Accurate prediction of patients’ ICU transfer events is of great significance for improving ICU treatment efficiency. ICU transition prediction task based on Electronic Health Records (EHR) is a temporal mining task like many other health informatics mining tasks. In the EHR-based temporal mining task, existing approaches are usually unable to mine and exploit patterns used to improve model performance. This article proposes a network based on Interval Pattern-Aware, Multi-Scale Interval Pattern-Aware (MSIPA) network. MSIPA mines different interval patterns in temporal EHR data according to the short, medium, and long intervals. MSIPA utilizes the Scaled Dot-Product Attention mechanism to query the contexts corresponding to the three scale patterns. Furthermore, Transformer will use all three types of contextual information simultaneously for ICU transfer prediction. Extensive experiments on real-world data demonstrate that an MSIPA network outperforms state-of-the-art methods.

The huge number of machine learning (ML) methods has resulted in significant information overload. Faced with an overwhelming number of ML methods, it is challenging to select appropriate ones for the given dataset and task. In general, the names of ML methods or datasets are rather condensed, thus lacking specific explanations, while the rich latent relationships between ML entities are not fully explored. In this paper, we propose a description-enhanced machine learning knowledge graph-based approach - DEKR - to help recommend appropriate ML methods for given ML datasets. The proposed knowledge graph (KG) not only includes the connections between entities but also contains the descriptions of the dataset and method entities. DEKR fuses the structural information with the description information of entities in the knowledge graph. It is a deep hybrid recommendation framework, which incorporates the knowledge graph-based and text-based methods, overcoming the limitations of previous knowledge graph-based recommendation systems that ignore the description information. There are two key components of DEKR: 1) a graph neural network aggregating information from multi-order neighbors with attention to enrich the seed (i.e. dataset or method) node's own representation, and 2) a deep collaborative filtering network based on the description text to obtain the linear and nonlinear interactions of description features. Through extensive experiments, we demonstrated the efficiency of DEKR, which outperforms the current state-of-the-art baselines by a large margin.

In China, with the continuous development of national health insurance policies, more and more people have joined the health insurance. How to accurately predict patients future medical treatment behavior becomes a hotspot issue. The biggest challenge in this issue is how to improve the prediction performance by modeling health insurance data with high-dimensional time characteristics. At present, most of the research is to solve this issue by using Recurrent Neural Networks (RNNs) to construct an overall prediction model for the medical visit sequences. However, RNNs can not effectively solve the long-term dependence, and RNNs ignores the importance of time interval of the medical visit sequence. Additionally, the global model may lose some important content to different groups. In order to solve these problems, we propose a Grouping and Global Attention based Time-aware Bidirectional Long Short-Term Memory (GGATB-LSTM) model to achieve medical treatment behavior prediction. The model first constructs a heterogeneous information network based on health insurance data, and uses a tensor CANDECOMP/PARAFAC decomposition method to achieve similarity grouping. In terms of group prediction, a global attention and time factor are introduced to extend the bidirectional LSTM. Finally, the proposed model is evaluated by using real dataset, and conclude that GGATB-LSTM is better than other methods.