The fusion of Hyperspectral image (HSI) and LiDAR data can dramatically enhance the classification accuracy, thus becoming a popular research topic. However, many studies are susceptible to the distribution difference across data collected by HSI and LiDAR sensors, resulting in misalignment of intermodal features and inadequate feature fusion. Therefore, a dual-branch information alignment fusion network (DIAFN) is proposed, aiming to exploit the highly correlated spatial information and complementary higher-order semantic features of multimodal data to achieve effective feature fusion. First, the dimension alignment module (DAM) is leveraged to extend the LiDAR data to align with HSI, and the weights of the LiDAR patches are adaptively learned using a dynamic adjustment approach to strengthen the complementary advantages between them. Second, a dual-branch information extraction module (IEM) is used to acquire complementary remote sensing features, and the distribution difference of multimodal data is reduced by optimizing alignment loss. Then, a dual-branch interactive fusion module (IFM) is proposed to fuse the acquired remote sensing features, and the intermodal relationship discriminator and channel discriminator are introduced to deeply mine and utilize the intrinsic connection between the features. Finally, the fused remote sensing features are classified for final category prediction. The model is experimented on two datasets to effectively demonstrate the superiority of DIAFN performance.

Hyperspectral images (HSIs) contain rich spatial and spectral information, while light detection and ranging (LiDAR) data can provide elevation details. Effectively fusing HSI and LiDAR data can help achieve more accurate classification results. However, the joint classification of HSI and LiDAR data still faces several challenges, such as the redundancy of HSI spectral bands, the limitations of singular multimodal data fusion strategy, and the high cost of pixelwise labeling in remote sensing images. To tackle these challenges, we propose a classification method based on candidate pseudo labels pruning and dual mixture of experts (CPLP-DMoEs). First, we employ the multihead mixture of bands (MMoBs) to perform diverse, dense mixing of spectral bands, thereby alleviating the issue of high similarity between adjacent bands. Then, to overcome the limitations of single fusion strategies, we design a mixture of multimodal fusion expert (MoMFE) mechanism, which selects and mixes multiple fusion experts (FEs) to achieve diverse feature fusion of HSI and LiDAR data. Next, we introduce information entropy to balance the selection of FEs. Finally, facing the challenge of limited labeled samples, we propose a candidate pseudo labels pruning (CPLP)-based semi-supervised learning method. CPLP can prune the candidate pseudo label set from both intrasample and intersample perspectives to obtain more reliable pseudo labels, thereby facilitating the learning of a more accurate classification model. The experimental results on three datasets, including Houston 2013, MUUFL, and Augsburg, validate the effectiveness of the proposed method.

The pretrained vision-language models (VLMs) have achieved outstanding performance in various visual tasks, primarily due to the knowledge they have acquired from massive image-text pairs. This enables VLMs to generalize to a wide range of downstream tasks. This article presents the first attempt to adapt VLMs for the joint classification task of hyperspectral image (HSI) and LiDAR data, aiming to leverage the well-learned VLMs to extract more generalizable features from diverse remote sensing image sources. Initially, using a patch encoder (PE), low-dimensional patches of HSI and LiDAR data are transformed into high-dimensional latent feature representations, meeting the dimensional requirements of VLMs for visual input data. Unlike traditional classifiers that rely on discrete class labels, VLM-based classification methods depend on continuous vectors, which can be derived from textual templates with class names, i.e., prompts. The classification performance of VLM-based methods heavily relies on these prompts, but prompt engineering not only demands extensive expert knowledge but also is extremely time-consuming. To address this, prompt tuning (PT) methods are introduced to enhance the generalizability of VLMs by adding spectral-based prompts to the vision encoder and incorporating randomly initialized, learnable text prompts (TPs) into the text encoder. Finally, through a novel class-discriminative loss function, the distance between text features of different classes is increased, thereby enhancing the model’s discriminative ability. Experimental results on the Houston 2013, Trento, and MUUFL datasets demonstrate that the proposed method can achieve competitive classification accuracy with a limited number of labeled pixels.

Labeling hyperspectral images (HSIs) is time-consuming and labor-intensive for researchers, so the deficiency of adequate labeling samples is a giant obstacle to conducting HSI classification. Especially, such issue is exacerbated when there are no available labeled samples in the target scene. For the sake of resolving aforesaid issue, we put forward a novel cross-scene HSI classification method namely bi-classifier adversarial augmentation network (BCAN) so as to transfer knowledge from a similar but different source domain to an unlabeled target domain. First, the source and target domain distributions are aligned by maximizing and minimizing the decision discrepancy between two classifiers, respectively. Then, more accurate samples corresponding to pseudo-labels are selected as reliable samples and added to the training set. Finally, the spectral band random zeroing (SBRZ) method is proposed to expand the training samples for reliable samples, which handles the problem of insufficient network training resulted from insufficient samples in the source domain. By using multi-classifiers for domain adaptation and data augmentation, the accuracy of the network for cross-scene HSI classification tasks are improved. BCAN can extract the source domain’s helpful information to complete the target domain classification task. Experiments conducted on ten HSI data pairs show that BCAN outperforms many state-of-the-art baselines.

Adversarial learning-based unsupervised hyperspectral image (HSI) classification methods usually adapt probability distributions by minimizing the statistical distance between similar pixels of different HSIs. Since the adversarial learning may weaken the discriminability of features, the extracted features will contain a lot of non-discriminative information, pixels with similar features may be classified as different classes. Therefore, directly reducing the statistical distance between similar pixels in a latent space may aggravate misclassification. To this end, we propose an unsupervised HSI classification method called soft instance-level domain adaptation with virtual classifier. First, the domain-invariant features of HSI are extracted by a graph convolutional network. Then, a feature similarity metric-based virtual classifier is constructed to output class probabilities of target-domain samples. Furthermore, to enable similar features of HSIs from different domains to be classified into the same class, the divergence between the real and virtual classifiers is reduced by minimizing the real and virtual classifier determinacy disparity. Finally, to reduce the influence of noisy pseudo-labels, a soft instance-level domain adaptation method is proposed. For each target-domain sample, the confidence coefficients are assigned to its corresponding positive and negative samples in the source domain, and a soft prototype contrastive loss is constructed and minimized to adapt two domains in an instance-level way. Experimental results on five real HSI datasets including Botswana, Kennedy Space Center, Pavia Center, Pavia University, and HyRANK demonstrate the effectiveness of our proposed method.

In existing generative adversarial networks (standard GAN and its variants), the discriminator is trained for recognizing the real data as positive while the generated data as negative. This kind of positive-negative classification criterion ignores the fact that the discriminator is a non-objective evaluator, which means that the image quality evaluated by the discriminator may fluctuate during the whole training progress. Considering this fact, we propose a novel GAN framework called Discriminator-Quality Evaluation GAN (DQE-GAN) by using the discriminator outputs to evaluate image quality. By dynamically classifying images into high discriminator-quality and low discriminator-quality samples, every adversarial iteration step can be more reasonable and objective. The convergence of DQE-GAN framework can be theoretically proved. Through extensive experiments, we demonstrate DQE-GANs’ ability of achieving better generated images faster and more stable.

Oriented object detection has achieved a significant progress in image processing. Compared with horizontal detection methods, oriented detectors add the orientation parameter in regression to locate objects. However, existing rotation and quadrilateral representations are not appropriate for oriented two-stage methods to generate efficient oriented proposals. In this paper, we propose a novel framework to detect oriented objects, termedshort side excursion detection(SSEDet). Inspired by the circle theorem, we propose a transformation method from horizontal rectangles to oriented ones to accurately describe oriented objects. To be specific, we exploit the offset of short sides relative to the top-right vertex to represent the orientation of rectangle. Compared with the horizontal rectangle, the representation parameters of oriented rectangle have only one more orientation parameter. Under the action of the orientation parameter, the one-to-one correspondence between representation parameters and oriented rectangle can be realized. Experimental results on commonly used datasets verify that SSEDet can generate high-quality oriented proposals.

A hyperspectral image (HSI) classification method named graph domain adversarial network with dual-weighted pseudo-label loss (GDAN-DWPL) is proposed in this letter. First, in order to extract more discriminative features, GDAN is applied to the transfer task of HSI. Then, a more reliable spectral–spatial graph is constructed by comprehensively utilizing the abundant spectral features and spatial contextual information. Finally, due to the misalignment of probability distribution on class-level caused by inaccurate pseudo-labels of target domain, a dual-weighted pseudo-label loss is proposed from the perspective of spatiality and confidence. By assigning larger weights to more reliable pixels and eliminating pixels with false pseudo-labels, the negative impact on learning process of prediction model can be reduced. Experimental results on four real HSI datasets show the superiority of GDAN-DWPL.

Due to the difficulty of labeling a large number of training samples of a hyperspectral image (HSI), unsupervised clustering methods have drawn great attention. The recently proposed broad learning (BL) can implement both linear and nonlinear mappings. However, the original BL is a supervised model. In this letter, a novel method named unsupervised BL (UBL) is introduced for HSI clustering. First, a graph-regularized sparse autoencoder is performed on the input and mapped feature of UBL in order to maintain the intrinsic manifold structure of origin HSI. Then, the objective function of UBL composed of an ℓ2-norm of output-layer weights and a graph regularization term is designed, which can be easily solved by choosing eigenvectors corresponding to the smallest eigenvalues. Finally, the HSI clustering results can be obtained by applying spectral clustering on the output of UBL. Experiments on three popular real HSI data sets demonstrate that, compared with several competitive methods, UBL can achieve better clustering performance.

Tumor clustering is a powerful approach for cancer class discovery which is crucial to the effective treatment of cancer. Many traditional clustering methods such as NMF-based models, have been widely used to identify tumors. However, they cannot achieve satisfactory results. Recently, subspace clustering approaches have been proposed to improve the performance by dividing the original space into multiple low-dimensional subspaces. Among them, low rank representation is becoming a popular approach to attain subspace clustering. In this paper, we propose a novel Low Rank Subspace Clustering model via Discrete Constraint and Hypergraph Regularization (DHLRS). The proposed method learns the cluster indicators directly by using discrete constraint, which makes the clustering task simple. For each subspace, we adopt Schatten -norm to better approximate the low rank constraint. Moreover, Hypergraph Regularization is adopted to infer the complex relationship between genes and intrinsic geometrical structure of gene expression data in each subspace. Finally, the molecular pattern of tumor gene expression data sets is discovered according to the optimized cluster indicators. Experiments on both synthetic data and real tumor gene expression data sets prove the effectiveness of proposed DHLRS.