Recently, unsupervised domain adaptation (UDA) techniques have been introduced for cross-scene hyperspectral image (HSI) classification tasks. These techniques aim to transfer knowledge from labeled source scenes to unlabeled target scenes, addressing the issue of limited supervisory information. However, most UDA methods fail to analyze the variability of domain shifts from different source samples to target ones, thus limiting the domain adaptation effect. To this end, this paper develops a consistency-aware customized learning (CACL) approach for cross-scene HSI classification. Overall, domain-level and class-level distribution alignment are designed separately. The former is implemented by adversarial training between the feature extractor and the domain discriminator. For the latter, the spectral-spatial prototypes of the source and target domains are first dynamically extracted, respectively. Then the prototype-based labels are assigned to the target domain samples, according to the cosine similarity-based cross-domain category prototype matching strategy. Considering that the consistency of the prototype-based labels with the predicted pseudo-labels reflects the degree of domain shifts of the target samples, a customized learning strategy is developed via inter-/intra-domain contrastive learning. With the joint domain-level and fine-grained class-level distribution alignment, the supervised information from the source domain is better migrated to the target domain, improving classification performance. Comprehensive experiments on two single-modal and one multi-modal cross-scene datasets demonstrate the effectiveness of the proposed algorithm.

Hyperspectral image (HSI) encompasses abundant spatial and spectral details, while Light Detection and Ranging (LiDAR) delivers precise elevation data. The amalgamation of HSI and LiDAR data significantly improves the precision of image classification. However, most methods focus solely on spatial features while neglecting frequency domain information, limiting the ability of deep models to characterize land cover. Furthermore, how to establish a sufficient interaction between different modalities is also an important issue. In this paper, we propose a novel multiscale cross-domain fusion network (MCFNet) for joint classification of HSI and LiDAR data. The main idea is that the wavelet transform can provide details at different resolutions simultaneously, supplementing spatial domain information and enriching feature representation. In addition, the multimodal fusion module (MFM) guided by HSI and the cross-domain fusion module (CDFM) strategy are developed to integrate features from diverse modalities and domains, respectively. Specifically, frequency domain features are extracted by discrete wavelet transform, and spatial domain features of the image are captured through a set of convolution operations. Then interactive fusion is performed by MFM and CDFM, and finally the integrated features are categorized using a classification module. Extensive experiments on three widely-used HSI and LiDAR datasets indicate that MCFNet outperforms the SOTA methods. The code will be available at https://github.com/MSFLabX/MCFNet.

Facing the pervasive problem of missing supervised information, cross-scene hyperspectral image (HSI) classification tasks based on unsupervised domain adaptation (UDA) techniques have emerged. However, lacking an integral view of feature-level alignment and decision-level analysis, most UDA methods treat target data with differential domain shift equally. Focused on this problem, a novel consistency-aware customized learning (CACL) approach is proposed, in this paper. Specifically, we develop a convolution-based domain-invariant feature learning network. First, the feature extractor is employed to extract spectral-spatial category prototypes. At the same time, domain-level distribution alignment is performed with the domain discriminator. Then, a customized learning strategy, i.e., inter/intra-domain contrast learning, is designed based on whether the pseudo-labels are consistent with the spectral-spatial prototype matchability labels. In addition, focal loss is introduced for information mining of hard samples. The experimental results demonstrate the state-of-the-art of the method.

How to learn the most valuable and useful features in convolutional neural networks (CNNs) is the key for accurate hyperspectral image classification (HSIC). Focused on this issue, we developed a local–global gated CNN (LGG-CNN), in this letter. The core is the simultaneous construction of local and global gated convolution blocks, with the aim to select highly discriminative information and filtering redundant information in hyperspectral images (HSIs). Different from traditional CNN methods treating all spectral–spatial features equally, the gated convolutions help in learning a normalized soft mask to guide the network to focus on valid features and neglect the invalid ones. Here, based on the CNN backbone, multilayer local features are first learned via gated convolutional architecture, which mainly consists of convolution operators and nonlinearly activation functions. At the same time, a global gated block (GGB) is designed to conduct feature serialization-mapping-patching operations, to learn global features from deeper layers with larger receptive fields. As a result, the local/GGBs can dynamically learn discriminative feature selection mechanisms for each channel at each spatial location. Then, the local and global features are fused at both the feature-level and decision-level. In this manner, the effective fusion of features by the multilayer LGG convolution blocks enables spatial interaction across layers, leading to further improvement in classification accuracy. Extensive experiments on three benchmark HSIC datasets demonstrate the superiority of LGG-CNN over some state-of-the-art methods. The source code of the proposed method is available athttps://github.com/Ding-Kexin/LGG-CNN.

Semi-supervised classification of remote sensing hyperspectral image (HSI) aims at exploiting both labeled and unlabeled samples for accurate land cover recognition. However, imbalanced data distribution and different classification difficulties negatively affect classification performance. Focused on this, a novel dual-stream class-adaptive network (DSCA-Net) is proposed for semi-supervised HSI classification, in this paper. First, a superpixel-guided label propagation module is introduced to alleviate the negative effect of imbalanced data distribution. Specifically, approximate estimation of labels for unlabeled samples is achieved via superpixel-wise similarity measure and label propagation, so that equal sampling is applied to each class. Then, a consistency regularization-based dual-stream network is constructed, which shares the same encoder for feature representation of either labeled or unlabeled samples. Based on this, two distinct classifiers are designed to force similar predictions can be achieved for various perturbed versions of the same unlabeled sample, thereby allowing unlabeled samples to train the model in a supervised manner. Finally, since different classes always have various degrees of learning difficulty, equal treatment may lead to overfitting of “easy” classes and biased prediction of “hard” classes. Unlike the traditional selection of unlabeled samples with a fixed threshold, dynamic class-adaptive thresholds are calculated according to the learning status of the model. In this manner, a higher threshold is assigned to “easy” classes to reduce sample redundancy, and a lower threshold is set for “hard” classes to select more samples. Experiment results demonstrate the effectiveness and superiority of the proposed method. Codes are available at https://github.com/luting-hnu/DSCA-Net.

Recently, deep learning presents a promising performance in the joint classification of multimodal remote sensing (RS) data. However, most of the approaches adopt a supervised learning manner, where the discrimination capability is limited by the paucity of labeled samples. Though some attempts have been made to develop semi-supervised methods, they prefer to select the highly confident predictions as pseudo ground truth and discard those unreliable ones. Actually, unreliable samples can also provide useful information, e.g., indicating the categories to which samples may belong and definitely not belong. Focused on this, a novel uncertainty-aware contrastive learning (UACL) method is proposed. Here, label uncertainty analysis based on multi-level probability estimation is first conducted to separate reliable and unreliable samples, which are then processed with a designed hybrid (“hard” or “soft”) contrastive learning (CL) strategy. For reliable samples, the “hard” CL pushes the network to learn features that will minimize the intra-class distance while maximizing the inter-class distance, according to the pseudo-labels. For unreliable samples, the “soft” CL aims to learn the similarity and difference among samples, where the predicted class probabilities are queried to estimate a soft mask for an adaptive feature similarity measurement. Moreover, a multimodal spectral-spatial joint feature representation pipeline of triple branches, i.e., one spectral branch for hyperspectral images (HSIs) and two spatial branches for multimodal data, is also introduced. By jointly learning from both labeled and unlabeled samples, more discriminative spectral-spatial feature representation will lead to a further boost in classification performance. Extensive experiments on four well-known multimodal datasets prove the effectiveness of the proposed semi-supervised classification method. Codes are available at https://github.com/Ding-Kexin/UACL.

Hyperspectral images (HSI) contain rich spatial and spectral detail information, while light detection and ranging (LiDAR) data can provide the elevation information. Thus, the fusion of HSI and LiDAR data can help for more accurate image classification, which becomes a hot research topic. However, it is difficult to capture complex local and global spatial-spectral associations, meanwhile, how to build an effective interaction between multi-modal data is another important issue. To this end, a novel global-local transformer network (GLT-Net) is proposed for the joint classification of HSI and LiDAR data, in this paper. The main idea is to fully exploit the advantage of the convolution operator in characterizing locally correlated features and the promising capability of transformer architecture in learning long-range dependencies. Moreover, multi-scale feature fusion and probabilistic decision fusion strategies are also designed in one framework, in order to further improve classification performance. Here, the proposed GLT-Net mainly consists of multi-scale local spatial feature learning, global spectral feature learning, and global-local feature fusion classification. In specific, multi-modal image cubes of different sizes are firstly extracted and sent into convolutional neural networks (CNNs) to learn local spatial features, which is followed by multi-modal information propagation and spatial-attention guided multi-scale feature fusion. Afterwards, by considering spectral feature channels from a sequential perspective, vision transformers are introduced to model the global spectral dependencies. Finally, multiple class estimations based on local and global features are integrated via a probabilistic decision fusion strategy. In this way, complementary information of multi-modal data as well as local/global spectral-spatial information can be fully mined and jointly utilized. Extensive experiments on three popular HSI and LiDAR datasets demonstrate that the proposed method performs superiority over state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/GLT-Net.

In recent years, deep learning presents a promising performance in hyperspectral image (HSI) classification, due to the powerful capability of automatically learning deep semantic characteristics of images. However, it is still difficult to learn highly discriminative features when limited samples are available for training a deep network. Focused on this issue, a novel end-to-end supervised contrastive learning network (SCL-Net) for spectral-spatial classification is proposed, in this paper. Instead of learning features of the individual sample, the supervised contrastive learning is introduced to capture the similarity and dissimilarity distribution properties of sample pairs in feature representation space. In this way, the need for plenty of training samples will be alleviated while an effective network training mechanism is provided for learning highly separative features. Here, the SCL-Net mainly consists of one pair-wise contrastive learning (PCL) sub-network and one multi-level spectral-spatial information fusion (MLSIF) sub-network. For the PCL sub-network, spectral vectors are projected into deep spectral features based on convolutional operators, which are then followed by distance evaluation between “positive” pairs of similar samples and “negative” pairs of dissimilar ones. Then, a spectral distance matrix is constructed to push the network to gradually learn better features of higher intra-class compactness and inter-class dispersion. For the MLSIF sub-network, a hybrid feature-decision fusion strategy is designed, where spatial and spectral features are jointly exploited to further boost classification performance. In specific, the feature fusion is conducted by connecting low/mid/high-level spectral and spatial features via weighting, while multiple class estimations based on multi-level fusion features are adaptively integrated via probabilistic decision fusion. Overall, these two sub-networks are collaboratively trained in one framework, by optimizing a defined joint loss function consisting of a contrastive loss and a cross-entropy loss. Compared with several state-of-the-art methods, the proposed method yields a superior classification performance in terms of both objective metrics and visual performance.