Soft prompt learning methods are effective for adapting vision-language models (VLMs) to downstream tasks. Nevertheless, empirical evidence reveals that existing methods tend to overfit seen classes and exhibit degraded performance on unseen classes. This limitation is due to the inherent bias in the training data towards the seen classes. To address this issue, we propose a novel soft prompt learning method, named Mixture-of-Prompts Distillation (MoPD), which can effectively transfer useful knowledge from hard prompts manually hand-crafted (a.k.a. teacher prompts) to the learnable soft prompt (a.k.a. student prompt), thereby enhancing the generalization ability of soft prompts on unseen classes. Moreover, the proposed MoPD method utilizes a gating network that learns to select hard prompts used for prompt distillation. Extensive experiments demonstrate that the proposed MoPD method outperforms state-of-the-art baselines, especially on unseen classes.

Message-passing graph neural networks (MPNNs) and structural features (SFs) are cornerstones for the link prediction task. However, as a common and intuitive mode of understanding, the potential of visual perception has been overlooked in the MPNN community. For the first time, we equip MPNNs with vision structural awareness by proposing an effective framework called Graph Vision Network (GVN), along with a more efficient variant (E-GVN). Extensive empirical results demonstrate that with the proposed frameworks, GVN consistently benefits from the vision enhancement across seven link prediction datasets, including challenging large-scale graphs. Such improvements are compatible with existing state-of-the-art (SOTA) methods and GVNs achieve new SOTA results, thereby underscoring a promising novel direction for link prediction.

The pretrained vision-language models (VLMs) have achieved outstanding performance in various visual tasks, primarily due to the knowledge they have acquired from massive image-text pairs. This enables VLMs to generalize to a wide range of downstream tasks. This article presents the first attempt to adapt VLMs for the joint classification task of hyperspectral image (HSI) and LiDAR data, aiming to leverage the well-learned VLMs to extract more generalizable features from diverse remote sensing image sources. Initially, using a patch encoder (PE), low-dimensional patches of HSI and LiDAR data are transformed into high-dimensional latent feature representations, meeting the dimensional requirements of VLMs for visual input data. Unlike traditional classifiers that rely on discrete class labels, VLM-based classification methods depend on continuous vectors, which can be derived from textual templates with class names, i.e., prompts. The classification performance of VLM-based methods heavily relies on these prompts, but prompt engineering not only demands extensive expert knowledge but also is extremely time-consuming. To address this, prompt tuning (PT) methods are introduced to enhance the generalizability of VLMs by adding spectral-based prompts to the vision encoder and incorporating randomly initialized, learnable text prompts (TPs) into the text encoder. Finally, through a novel class-discriminative loss function, the distance between text features of different classes is increased, thereby enhancing the model’s discriminative ability. Experimental results on the Houston 2013, Trento, and MUUFL datasets demonstrate that the proposed method can achieve competitive classification accuracy with a limited number of labeled pixels.

Adversarial learning-based unsupervised hyperspectral image (HSI) classification methods usually adapt probability distributions by minimizing the statistical distance between similar pixels of different HSIs. Since the adversarial learning may weaken the discriminability of features, the extracted features will contain a lot of non-discriminative information, pixels with similar features may be classified as different classes. Therefore, directly reducing the statistical distance between similar pixels in a latent space may aggravate misclassification. To this end, we propose an unsupervised HSI classification method called soft instance-level domain adaptation with virtual classifier. First, the domain-invariant features of HSI are extracted by a graph convolutional network. Then, a feature similarity metric-based virtual classifier is constructed to output class probabilities of target-domain samples. Furthermore, to enable similar features of HSIs from different domains to be classified into the same class, the divergence between the real and virtual classifiers is reduced by minimizing the real and virtual classifier determinacy disparity. Finally, to reduce the influence of noisy pseudo-labels, a soft instance-level domain adaptation method is proposed. For each target-domain sample, the confidence coefficients are assigned to its corresponding positive and negative samples in the source domain, and a soft prototype contrastive loss is constructed and minimized to adapt two domains in an instance-level way. Experimental results on five real HSI datasets including Botswana, Kennedy Space Center, Pavia Center, Pavia University, and HyRANK demonstrate the effectiveness of our proposed method.

A hyperspectral image (HSI) classification method named graph domain adversarial network with dual-weighted pseudo-label loss (GDAN-DWPL) is proposed in this letter. First, in order to extract more discriminative features, GDAN is applied to the transfer task of HSI. Then, a more reliable spectral–spatial graph is constructed by comprehensively utilizing the abundant spectral features and spatial contextual information. Finally, due to the misalignment of probability distribution on class-level caused by inaccurate pseudo-labels of target domain, a dual-weighted pseudo-label loss is proposed from the perspective of spatiality and confidence. By assigning larger weights to more reliable pixels and eliminating pixels with false pseudo-labels, the negative impact on learning process of prediction model can be reduced. Experimental results on four real HSI datasets show the superiority of GDAN-DWPL.