Traffic prediction provides vital support for urban traffic management and has received extensive research interest. By virtue of the ability to effectively learn spatial and temporal dependencies from a global view, Transformers have achieved superior performance in long-term traffic prediction. However, existing methods usually underrate the complex spatio-temporal entanglement in long-range sequences. Compared with purely temporal entanglement, spatio-temporal data emphasizes the entangled dynamics under the restrictions of traffic networks, which brings additional difficulties. Moreover, the computational costs of spatio-temporal Transformers scale quadratically as the sequence length grows, limiting their applications on long-range and large-scale scenarios. To address these problems, we propose a decomposed spatio-temporal Mamba (DST-Mamba) for traffic prediction. We aim to apply temporal decomposition to the entangled sequences and obtain the seasonal and trend parts. Shifting from the temporal view to the spatial view, we leverage Mamba, a state space model with near-linear complexity, to capture seasonal variations in a node-centric manner. Meanwhile, multi-scale trend information is extracted and aggregated by simple linear layers. Such combination equips DST-Mamba with superior capability to model long-range spatio-temporal dependencies while remaining efficient compared with Transformers. Experimental results across five real-world datasets demonstrate that DST-Mamba can capture both local fluctuations and global trends within traffic patterns, achieving state-of-the-art performance with favorable efficiency.

Spatio-temporal neural networks have been successfully applied to weather forecasting tasks recently. The key notion is to learn spatio-temporal features concurrently from spatial and temporal dependencies. Existing methods are mainly based on local smoothness assumptions where the features are learned by accumulating information in local spatio-temporal regions. However, the weather conditions in a certain spatio-temporal region are usually influenced by global meteorological changes and long-range historical weather conditions. Therefore, these methods that ignore the large-scale spatio-temporal effects can hardly learn effective features. In this paper, we propose a novel spatio-temporal Transformer network in weather forecasting to address the above challenges. The main idea is to leverage the Transformer architecture to carefully capture the multi-scale spatial and long-range temporal information in weather data. First, we propose to combine the global and local position encodings based on absolute geographic locations and relative geodesic distances and insert them into the spatial Transformer to extract the multi-scale spatial information in meteorological graphs. Then, we further capture the long-range temporal dependencies by a temporal Transformer where the attention mechanism is used to improve the representation ability and scalability of the models. Extensive experiments over real weather datasets demonstrate the effectiveness of our framework.

Weather forecasting has become a popular research topic recently, which mainly benefits from the development of spatio-temporal neural networks to effectively extract useful patterns from weather data. Generally, the weather changes in the meteorological system are governed by physical principles. However, it is challenging for spatio-temporal methods to capture the physical knowledge of meteorological dynamics. To address this problem, we propose in this paper a spatio-temporal Transformer network with physical knowledge distillation (PKD-STTN) for weather forecasting. First, the teacher network is implemented by a differential equation network that models weather changes by the potential energy in the atmosphere to reveal the physical mechanism of atmospheric movements. Second, the student network uses a spatio-temporal Transformer that concurrently utilizes three attention modules to comprehensively capture the semantic spatial correlation, geographical spatial correlation, and temporal correlation from weather data. Finally, the physical knowledge of the teacher network is transferred to the student network by inserting a distillation position encoding into the Transformer. Notice that the output of the teacher network is distilled to the position encoding rather than the output of the student network, which can largely utilize physical knowledge without influencing the feature extraction process of Transformers. Experiments on benchmark datasets show that the proposed method can effectively utilize physical principles of weather changes and has obvious performance advantages compared with several strong baselines.

Learning brain effective connectivity networks (ECN) from functional magnetic resonance imaging (fMRI) data has gained much attention in recent years. With the successful applications of deep learning in numerous fields, several brain ECN learning methods based on deep learning have been reported in the literature. However, current methods ignore the deep temporal features of fMRI data and fail to fully employ the spatial topological relationship between brain regions. In this article, we propose a novel method for learning brain ECN based on spatiotemporal graph convolutional models (STGCM), named STGCMEC, in which we first adopt the temporal convolutional network to extract the deep temporal features of fMRI data and utilize the graph convolutional network to update the spatial features of each brain region by aggregating information from neighborhoods, which makes the features of brain regions more discriminative. Then, based on such features of brain regions, we design a joint loss function to guide STGCMEC to learn the brain ECN, which includes a task prediction loss and a graph regularization loss. The experimental results on a simulated dataset and a real Alzheimer's disease neuroimaging initiative (ADNI) dataset show that the proposed STGCMEC is able to better learn brain ECN compared with some state-of-the-art methods.

Due to the extraordinary abilities in extracting complex patterns, graph neural networks (GNNs) have demonstrated strong performances and received increasing attention in recent years. Despite their prominent achievements, recent GNNs do not pay enough attention to discriminate nodes when determining the information sources. Some of them select information sources from all or part of neighbors without distinction, and others merely distinguish nodes according to either graph structures or node features. To solve this problem, we propose the concept of the Influence Set and design a novel general GNN framework called the graph influence network (GINN), which discriminates neighbors by evaluating their influences on targets. In GINN, both topological structures and node features of the graph are utilized to find the most influential nodes. More specifically, given a target node, we first construct its influence set from the corresponding neighbors based on the local graph structure. To this aim, the pairwise influence comparison relations are extracted from the paths and a HodgeRank-based algorithm with analytical expression is devised to estimate the neighbors' structure influences. Then, after determining the influence set, the feature influences of nodes in the set are measured by the attention mechanism, and some task-irrelevant ones are further dislodged. Finally, only neighbor nodes that have high accessibility in structure and strong task relevance in features are chosen as the information sources. Extensive experiments on several datasets demonstrate that our model achieves state-of-the-art performances over several baselines and prove the effectiveness of discriminating neighbors in graph representation learning.

Spatiotemporal graph neural networks (GNNs) have been used successfully in traffic prediction in recent years, primarily owing to their ability to model complex spatiotemporal dependencies within irregular traffic networks. However, the feature extraction processes in these methods are limited in their exploration of the inner properties of traffic data. Specifically, graph and temporal convolutions are local operations and can hardly utilize information from wider ranges, which may affect the long-term prediction performance of such methods. Furthermore, deep spatiotemporal GNNs easily suffer from poor generalization owing to overfitting. To address these problems, this study presents a novel traffic prediction method that integrates self-supervised learning and self-distillation into spatiotemporal GNNs. First, a self-supervised learning module is used to explore the knowledge from the input data. An auxiliary task based on temporal continuity is designed to capture the contextual information in traffic data. Second, a self-distillation framework is developed as an implicit regularization approach that transfers knowledge from the model itself. The combination of self-supervision and self-distillation further mines the knowledge from the data and the model, and the generalization ability and stability of the prediction model can be improved. The proposed model achieved superior or competitive results compared with several strong baselines on six traffic prediction datasets. In particular, the maximum performance improvement ratios for the six datasets were 3.0% (MAE), 5.2% (RMSE), and 3.8% (MAPE). These results demonstrate the effectiveness of the proposed method.

Recent interests in graph neural networks (GNNs) have received increasing concerns due to their superior ability in the network embedding field. The GNNs typically follow a message passing scheme and represent nodes by aggregating features from neighbors. However, the current aggregation methods assume that the network structure is static and define the local receptive fields under visible connections, which consequently fails to consider latent or high-order structures. Besides, the aggregation methods are known to have a depth dilemma due to the over-smoothness issues. To solve the above shortcomings, we present in this article a compact graph convolutional network framework which defines the graph receptive fields based on diffusion paths and explicitly compresses the neural networks with sparsity regularization. The proposed model seeks to learn from invisible connections and recover the latent proximity. First, we infer the high-order proximity and construct diffusion paths by diffusion samplings. Compared with random walk samplings, the diffusion samplings are based on regions instead of paths. The network inference then obtains accurate weights that can be leveraged to build small but informative receptive fields with salient neighbors. Second, to utilize the deep information while avoiding overfitting, we propose learning a lightweight model by introducing a nonconvex regularizer. Numerical comparisons with the existing network embedding methods under unsupervised feature learning and supervised classification show the effectiveness of our model.

Discrete network embedding emerged recently as a new direction of network representation learning. Compared with traditional network embedding models, discrete network embedding aims to compress model size and accelerate model inference by learning a set of short binary codes for network vertices. However, existing discrete network embedding methods usually assume that the network structures (e.g., edge weights) are readily available. In real-world scenarios such as social networks, sometimes it is impossible to collect explicit network structure information and it usually needs to be inferred from implicit data such as information cascades in the networks. To address this issue, we present an end-to-end discrete network embedding model for latent networks DELN that can learn binary representations from underlying information cascades. The essential idea is to infer a latent Weisfeiler-Lehman proximity matrix that captures node dependence based on information cascades and then to factorize the latent Weisfiler-Lehman matrix under the binary node representation constraint. Since the learning problem is a mixed integer optimization problem, an efficient maximal likelihood estimation based cyclic coordinate descent (MLE-CCD) algorithm is used as the solution. Experiments on real-world datasets show that the proposed model outperforms the state-of-the-art network embedding methods.

Network embedding is a promising topic that maps the vertices to the latent space while keeps the structural proximity in the original space. The network embedding task is difficult since the network vertices have no specific time or space orders. Models that used to extract information from images and texts with regular space or time structures can not be directly applied in network heading. The key feature of network embedding methods should be further exploited. Previous network embedding reviews mainly focus on the models and algorithms used in different methods. In this survey, we review the network embedding works in the stochastic perspective either in data side or model side. Roughly, the network embedding methods fall into three main categories: matrix based methods, random walk based methods and aggregated based methods. We focus on the applications of stochastic models in solving the challenges of network embedding in data processing and modeling following the line of the three categories.