Hyperspectral camouflaged object tracking remains a significant challenge due to the high similarity between objects and replicas in texture and color. Despite recent progress, the bias present in the tracker and the embedding token hinders the model training. Specifically, most methods rely on false-color three-channel images to fine-tune RGB-based trackers. However, it introduces a confounding effect within the RGB domain, potentially leading to harmful biases that misguide the model toward spurious correlations while neglecting the critical spectral discrimination inherent in hyperspectral images. Furthermore, current token-type embedding methods overlook the key correlations between templates and searches, ultimately confusing correlation and impairing tracking performance. To address these challenges, this paper proposes a new unbiased tracking framework named Causal HyperPrompter. It first introduces a structural causal model to disentangle and control exclusive causal factors during tracking, and incorporates a counterfactual intervention strategy to eliminate confounding variables and mitigate the bias inherited from RGB-based models. In addition, we present a novel token-type embedding module that integrates local spectral angle modeling to enhance the semantic link between template and search tokens, thereby improving the model's sensitivity to object localization. Lastly, to overcome the difficulty of manually initializing the bounding box and addressing data scarcity, we introduce a large-scale hyperspectral camouflaged object detection and tracking dataset, BihoT-130 k, consisting of 1,30,750 annotated frames across various camouflage scenes. Extensive experiments on multiple large-scale datasets illustrate the effectiveness of our proposed methods.

Hyperspectral video (HSV) offers valuable spatial, spectral, and temporal information simultaneously, making it highly suitable for handling challenges such as background clutter and visual similarity in object tracking. However, existing methods primarily focus on band regrouping and rely on RGB trackers for feature extraction, resulting in limited exploration of spectral information and difficulties in achieving complementary representations of object features. In this paper, a spatial-spectral fusion network with spectral angle awareness (SSF-Net) is proposed for hyperspectral (HS) object tracking. Firstly, to address the issue of insufficient spectral feature extraction in existing networks, a spatial-spectral feature backbone ( $S^{2}$ FB) is designed. With the spatial and spectral extraction branch, a joint representation of texture and spectrum is obtained. Secondly, a spectral attention fusion module (SAFM) is presented to capture the intra- and inter-modality correlation to obtain the fused features from the HS and RGB modalities. It can incorporate the visual information into the HS context to form a robust representation. Thirdly, to ensure a more accurate response to the object position, a spectral angle awareness module (SAAM) is designed to investigate the region-level spectral similarity between the template and search images during the prediction stage. Furthermore, a novel spectral angle awareness loss (SAAL) is developed to offer guidance for the SAAM based on similar regions. Finally, to obtain the robust tracking results, a weighted prediction method is considered to combine the HS and RGB predicted motions of objects to leverage the strengths of each modality. Extensive experiments on the HOTC-2020, HOTC-2024, and BihoT datasets demonstrate the effectiveness of the proposed SSF-Net compared with state-of-the-art trackers. The source code will be available at https://github.com/hzwyhc/hsvt.

Hyperspectral object tracking (HOT) has many important applications, particularly in scenes where objects are camouflaged. The existing trackers can effectively retrieve objects via band regrouping because of the bias in the existing HOT datasets, where most objects tend to have distinguishing visual appearances rather than spectral characteristics. This bias allows a tracker to directly use the visual features obtained from the false-color images generated by hyperspectral images (HSIs) without extracting spectral features. To tackle this bias, the tracker should focus on the spectral information when object appearance is unreliable. Thus, we provide a new task called hyperspectral camouflaged object tracking (HCOT) and meticulously construct a large-scale HCOT dataset, BihoT, consisting of 41912 HSIs covering 49 video sequences. The dataset covers various artificial camouflage scenes, where objects have similar appearances, diverse spectrums, and frequent occlusion (OCC), making it a challenging dataset for HCOT. Besides, a simple but effective baseline model, named spectral prompt-based distractor-aware network (SPDAN), is proposed, comprising a spectral embedding network (SEN), a spectral prompt-based backbone network (SPBN), and a distractor-aware module (DAM). Specifically, the SEN extracts spectral-spatial features via 3-D and 2-D convolutions to form a refined prompt representation. Then, the SPBN fine-tunes powerful RGB trackers with spectral prompts and alleviates the insufficiency of training samples. Moreover, the DAM utilizes a novel statistic to capture the distractor caused by occlusion from objects and background and corrects the deterioration of the tracking performance via a novel motion predictor. Extensive experiments demonstrate that our proposed SPDAN achieves the state-of-the-art performance on the proposed BihoT and other HOT datasets.

Hyperspectral videos (HSVs) offer not only spatial information but also diagnostic spectral features. Due to the fact that spectral features are only related to the material of the object, this advantage can address the issue of RGB video tracking failure when the object and background are visually similar. However, the effectiveness of deep learning models is limited due to insufficient HSV training data. Existing methods tend to divide a hyperspectral image (HSI) into several three-channel false-color images to leverage the existing RGB trackers for transfer learning. Nonetheless, these methods lack adequate exploration of band interrelations and overlook correlation among objects prior to similarity calculation. In this article, a transformer-based band regrouping and feature refinement network (TBR-Net) is introduced, which is specifically tailored for hyperspectral object tracking. To maximize the potential of the RGB tracker and enhance the use of available training data, we propose a transformer-based band regrouping (TBR) method. By modeling long-range spectral dependencies, the inherent context information among bands is captured, which is subsequently utilized to reorganize bands into several false-color images. Furthermore, to combine the relationship of the template and the search (T & S) frames into a correlation calculation, a feature refinement module (FRM) is designed. The cross-attention mechanism enables mutual relation modeling, allowing similar regions to be perceived and form discriminative feature representation. As a result, a hyperspectral tracker can be efficiently trained via transfer learning to address the data insufficiency challenge, while the mutual perception between objects further enhances the tracking performance. Its effectiveness is validated by extensive benchmark experiments, which demonstrate that the TBR-Net surpasses state-of-the-art methods.

Visible-infrared person re-identification (VI-ReID) is a challenging task in computer vision, aiming at matching people across images from visible and infrared modalities. The widely used VI-ReID framework consists of a convolution neural backbone network that extracts the visual features, and a feature embedding network to project heterogeneous features to the same feature space. However, many studies based on the existing pre-trained models neglect potential correlations between different locations and channels within a single sample during the feature extraction. Inspired by the success of the Transformer in computer vision, we extend it to enhance feature representation for VI-ReID. In this paper, we propose a discriminative feature learning network based on a visual Transformer (DFLN-ViT) for VI-ReID. Firstly, to capture long-term dependencies between different locations, we propose a spatial feature awareness module (SAM), which utilizes a single-layer Transformer with a novel patch-embedding strategy to encode location information. Secondly, to refine the representation at each channel, we design a channel feature enhancement module (CEM). The CEM treats the features of each channel as a sequence of Transformer inputs, taking advantage of the Transformer's ability to model long-term dependencies. Finally, we propose a Triplet-aided Hetero-Center (THC) loss to learn more discriminative feature representation by balancing the cross-modality distance and intra-modality distance of the center. The experimental results on two datasets show that our method can significantly improve the VI-ReID performance, outperforming most state-of-the-art methods.