Functional connectivity (FC) derived from fMRI reflects the interactions between brain regions of interest (ROIs). It has become one of the important features of individualized prediction. However, in studies using FC as input, some studies have extracted global features directly from the entire brain FC, lacking focus on local critical information. On the other hand, some studies have extracted local critical features from selected ROIs or connections, but lack access to global contextual information. In this paper, we propose a novel method, namely, Global-Local Brain Network (GLBN), focusing on both global contextual information and critical local information. We validate our proposed method on the largescale public dataset, the Cambridge Centre for Ageing and Neuroscience (Cam-CAN). For the prediction of age and fluid intelligence, GLBN achieves the mean absolute errors of 6.084, and 4.160, with Pearson’s correlations of 0.908, and 0.641, respectively. Our method demonstrates superior prediction accuracy compared to existing studies. Additionally, we visualize the brain ROIs that play crucial roles in the prediction tasks, affirming the biological interpretability of GLBN.

High spatial, high spectral, and high temporal ($\text {H}^{3}$) information of the objects of interest can be provided by hyperspectral video, which makes it possible to track objects in complex scenarios. However, during motion, changes in the target’s appearance, background, and spectral information can degrade the performance of existing hyperspectral trackers due to insufficient training data. Consequently, this results in weak generalization of these trackers. In this article, to solve the above problems, a deep temporal-spectral–spatial anchor-free Siamese tracking network for hyperspectral video object tracking, namely HA-Net, is proposed. In HA-Net, a Siamese spectral enhancement tracker module based on an RGB tracker (pseudo-color tracker) is designed, which uses the powerful feature expression capabilities of the deep network to learn more discriminative deep spectral features for identifying objects in complex scenarios. The pseudo-color tracker is introduced to solve the problem of model performance limitation due to insufficient training data. By introducing the temporal-spectral–spatial online discrimination learning module, the temporal-spectral–spatial information of the target can be dynamically modeled to adapt to new targets and the dynamic changes of targets. Benefiting from the double Siamese network architecture, the model can be effectively trained from scratch with less than 20 000 training samples. Online learning of temporal-spectral–spatial information for the target, particularly in cases of insufficient training data, can alleviate the issue of model degradation. This approach enhances the model’s robustness when tracking the target in complex scenes. In the 2021 IEEE WHISPERS Hyperspectral Object Tracking (HOT) Challenge, HA-Net obtained the best performance, with a distance precision (DP) score of 0.948 and an area under the curve (AUC) score of 0.688. The running speed is also 14 frames/s, which is superior to the existing hyperspectral object trackers for hyperspectral video. The source code is available athttps://github.com/zhenliuzhenqi/HOT.

Remote-sensing scene classification (RSSC) is crucial for remote-sensing image interpretation and has become a research hotspot in recent years. However, the high complexity of remote-sensing scenes causes most RSSC models to fail to accurately capture key objects, resulting in low classification accuracy. Meanwhile, it is intractable to effectively distinguish similar scenes, such as forest and meadow, whose semantic labels are mainly determined by wide-scale features. In addition, existing remote-sensing attention mechanisms are heuristic settings, which require expert knowledge and extensive experiments. To solve the above problems, a novel plug-and-play hyperparameter-free attention module (HFAM) based on feature map mathematical calculation is proposed in this work. HFAM uses statistical indicators to quantitatively characterize the fluctuations of feature maps that can accurately locate key features and distinguish different scenes, alleviating the problems of intraclass diversity and interclass similarity. Moreover, HFAM adaptively acquires attention weights by performing simple mathematical calculations on the feature maps, which solves the problem of difficult adjustment of hyperparameters. Our proposed HFAM can be expediently inserted into the existing ConvNet models without increasing the number of model’s parameters. Extensive contrast experiments with several famous plug-and-play attention modules on three mainstream datasets reveal the superiority of our HFAM in accuracy, number of parameters, and calculation amount. Moreover, compared with state-of-the-art methods, it also demonstrated considerable competitiveness.

Hyperspectral object tracking is aimed at tracking targets by using both the spatial information and abundant spectral information, overcoming the drawbacks of traditional RGB tracking in complex scenarios, such as the low resolution or background clutter. However, the current hyperspectral object tracking methods usually have a high computational complexity, due to the huge data volume, making them difficult to apply to real-time applications on edge devices (e.g., robots, unmanned aerial vehicles, and satellites) with limited computational resources. In this paper, a lightweight dual Siamese network for onboard hyperspectral object tracking—termed SiamOHOT—is proposed for real-time and onboard tracking. Specifically, a joint spatial-spectral knowledge distillation method is proposed to teach a lightweight dual Siamese tracker to learn from a deep tracker— SiamHYPER—so that the number of parameters can be compressed to improve the computational efficiency. In addition, a deep learning inference optimizer is introduced to fuse the layers with similar functions and quantify the parameters of the network, to further promote the processing speed when deployed on an embedded platform. The proposed lightweight model was verified using the 2021 WHISPERS Hyperspectral Object Tracking Challenge dataset, and achieved a superior efficiency and accuracy. In addition, a prototype system was built integrating a snapshot hyperspectral imager, the SiamOHOT tracking algorithm, and an artificial intelligence edge device (NVIDIA Jetson Xavier NX), to realize real-time imaging and tracking. The inference speed of the optimized SiamOHOT network is nearly doubled when compared to the teacher model on the prototype system.

Target tracking has received increased attention in the past few decades. However, most of the target tracking algorithms are based on RGB video data, and few are based on hyperspectral video data. With the development of the new “snapshot” hyperspectral sensors, hyperspectral videos can now be easily obtained. However, hyperspectral video target tracking datasets are still rare. In this article, a high spectral-spatial-temporal resolution hyperspectral video target tracking algorithm framework (H3Net) based on deep learning is proposed. The proposed framework consists of two main parts: 1) an unsupervised deep learning-based target tracking training framework for hyperspectral video; and 2) a dual-branch network structure based on a Siamese network. Using the dual-branch network, the H3Net framework can utilize both the spatial and spectral information. The combination of deep learning and a discriminative correlation filter (DCF) makes the features extracted by deep learning more suitable for the DCF. Compared with hyperspectral images, hyperspectral video data require more manpower to annotate, so we propose an unsupervised approach to train H3Net, without any annotation. To solve the problem of the lack of hyperspectral video datasets, we built a 25-band hyperspectral video dataset (the high spectral-spatial-temporal resolution hyperspectral video dataset: the WHU-Hi-H3dataset) for target tracking. The experimental results obtained with the WHU-Hi-H3dataset confirm the potential of unsupervised deep learning in hyperspectral video target tracking.

Satellite video is an important new earth observation data source that can be used to acquire large-scale dynamic information. Satellite video object segmentation (SVOS) is aimed at separating the foreground and background of satellite video and is a fundamental processing task for satellite video. To date, the state-of-the-art research into SVOS has mainly focused on unsupervised target extraction methods through hand-crafted features and post-processing operations, which are prone to obtaining incomplete contours of the targets and result in foreground aperture problem. Furthermore, the small size of targets and appearance deformation make the SVOS more difficult. Therefore, in this article, a spatio-temporal dual-branch network is proposed with predictive feature learning for the SVOS task. The proposed model consists of a temporal coherence branch and a spatial segmentation branch. In the temporal coherence branch, the Wasserstein generative adversarial network (WGAN) architecture is utilized for the future frame prediction to exploit temporal information, which captures the dynamic appearance and motion cues from the unlabeled satellite video data through predictive feature learning module in an adversarial manner. As a result, the proposed method can obtain segmentation results with temporal consistency, while avoiding the generation of optical flow images. In the spatial segmentation branch, a fully convolutional network (FCN) is used to extract the high-level spatial information of the satellite video and achieve end-to-end SVOS, without any post-processing operations. In the network implementation, boundary loss is used to solve the highly unbalanced segmentation problem caused by small size of the targets. The two branches of the network are also mutually constrained, to improve the final object segmentation results. The visual and quantitative results of three experiments all demonstrate that the proposed method outperforms the other current SVOS models.

Hyperspectral videos can provide the spatial, spectral, and motion information of targets, which makes it possible to track camouflaged targets that are similar to the background. However, hyperspectral object tracking is a challenging task, due to the huge hyperspectral video data dimension and the "data hungry" problem for the model training. Insufficient training data can seriously interfere with the accuracy and generalization of the tracking models. In this paper, a dual deep Siamese network framework for hyperspectral object tracking (SiamHYPER) is proposed for learning a hyperspectral tracker from a pretrained RGB tracker in the case of the "data hungry" problem. Specifically, in addition to a pretrained RGB-based Siamese tracker, a hyperspectral target-aware module is designed to mine the spectral information during the target prediction, and a spatial-spectral cross-attention module is introduced to further fuse the deep spatial and spectral features extracted from the RGB tracker and the hyperspectral target-aware module. Benefiting from the guidance training of the RGB tracker, a robust hyperspectral object tracker can be trained effectively with only a small number of hyperspectral video samples, to overcome the "data hungry" problem. In the experiments conducted in this study, the SiamHYPER framework was verified using SiamBAN and SiamRPN++, with 13 000 frames of hyperspectral videos for training, and achieved the best performance on the publicly available hyperspectral dataset released as part of the WHISPERS Hyperspectral Object Tracking Challenge. The area under the curve (AUC) of SiamHYPER was increased by nearly 8.9% and 7.2%, respectively, when compared with the current state-of-the-art RGB-based and hyperspectral trackers. In addition, the processing speed of SiamHYPER was 19 FPS, which is much higher than that of the current state-of-the-art hyperspectral trackers. The source code is available at zhenliuzhenqi/HOT: Hyperspectral object tracking (github.com).