The target classification of synthetic aperture radar (SAR) is an important technique of SAR image processing. Recently, many deep convolutional neural network (CNN)-based methods have been proposed for SAR target classification. However, feature extraction abilities of these CNN-based methods are insufficient. On the other hand, attention-based methods (e.g., swin Transformer) have the advantage of capturing the local-global features of images. Moreover, there always exists inherently noise in SAR images influenced by the process of emitted pulses, which hinders the improvement of accuracy for SAR target classification. To solve the both problems, this study explores a swin Transformer with improved blind-spot network (STr-BS) to alleviate the bad influence caused by speckle noise in SAR image and enhance the classification result. Specifically, the denoising process in STr-BS designs an improved blind-spot network in unsupervised setting without requiring the clean SAR images as input. Then, the outputs of the improved blind-spot network are as the input of the swin Transformer for the subsequent local-global feature extraction. The proposed STr-BS is tested on two public datasets (MSTAR and OpenSARShip), and the experimental results demonstrate the effectiveness of the proposed methods in comparison to other state-of-the-art approaches.

Radar jamming recognition is a fundamental step for anti-jamming techniques. However, the recognition performance is impeded in insufficient samples situation. Ensemble learning provides an effective way to address this issue. In this paper, first, a novel ensemble learning-based radar Transformer (i.e., RadarTR-E) is proposed to improve recognition performance with insufficient samples. Specifically, it votes on the predictions among sub-recognizers to increase recognition accuracy, where RadarTR is employed to effectively capture long-range dependencies. Then, a dynamic label smoothing method (i.e., RadarTR-E-DLS) is further proposed to mitigate overfitting. In detail, dynamic soft labels are designed to prevent overconfidence towards certain radar jamming types. Therefore, the proposed RadarTR-E-DLS achieves better recognition accuracy. Compared with other advanced methods, the experimental results show the superior recognition performance of the proposed methods.

With the increasing availability and openness of remote sensing (RS) data collected from diverse sensors, there has been a growing interest in multimodal RS data classification. Nowadays, in the area of deep learning, there is a paradigm shift with the rise of foundation models, which are trained on large-scale datasets and are adaptable to a wide range of downstream tasks. In this study, the potential and effectiveness of foundation models for multimodal RS data classification is investigated. The training datasets of foundation models and multimodal RS datasets are quite different, and therefore, it is difficult to use a pretrained foundation model for multimodal RS data classification directly. To mitigate this difficulty, this article proposes a foundation model adaptation (FMA) framework for multimodal RS data classification without fine-tuning the parameters. Specifically, two learnable modules, i.e., cross-spatial interaction module and cross-channel interaction module, are proposed to add to the foundation model for extracting multimodal-specific representations. The cross-spatial and cross-channel interaction modules extract the characteristics of unimodal features along the spatial dimension and channel dimension, respectively. To effectively tackle the disparities among various RS modalities, an alignment approach (FMA2) is further explored based on the FMA. The FMA2 describes dependencies between different modalities by establishing a coupling score function, which can further enhance classification performance. To demonstrate the effectiveness and superiority of the FMA framework, comprehensive experiments are conducted on three multimodal RS datasets, showing improvement over the advanced multimodal RS data classification image methods.

Recently, deep learning models have dominated hyperspectral image (HSI) classification. Nowadays, deep learning is undergoing a paradigm shift with the rise of transformer-based foundation models. In this study, the potential of transformer-based foundation models, including the vision foundation model (VFM) and language foundation model (LFM), for HSI classification are investigated. First, to improve the performance of traditional HSI classification tasks, a spectral-spatial VFM-based transformer (SS-VFMT) is proposed, which inserts spectral-spatial information into the pretrained foundation transformer. Specifically, a given pretrained transformer receives HSI patch tokens for long-range feature extraction benefiting from the prelearned weights. Meanwhile, two enhancement modules, i.e., spatial and spectral enhancement modules (SpaEMs$\backslash $SpeEMs), utilize spectral and spatial information for steering the behavior of the transformer. Besides, an additional patch relationship distillation strategy is designed for SS-VFMT to exploit the pretrained knowledge better, leading to the proposed SS-VFMT-D. Second, based on SS-VFMT, to address a new HSI classification task, i.e., generalized zero-shot classification, a spectral-spatial vision-language-based transformer (SS-VLFMT) is proposed. This task is to recognize novel classes not seen during training, which is more meaningful as the real world is usually open. The SS-VLFMT leverages SS-VFMT to extract spectral-spatial features and corresponding hash codes while integrating a pretrained language model to extract text features from class names. Experimental results on HSI datasets reveal that the proposed methods are competitive compared to the state-of-the-art methods. Moreover, the foundation model-based methods open a new window for HSI classification tasks, especially for HSI zero-shot classification.

Several fundamental remote sensing (RS) image processing tasks, including classification, segmentation, and detection, have been set to serve for manifold applications. In the RS community, the individual tasks have been studied separately for many years. However, the specialized models were only capable of a single task. They lacked the adaptability for generalizing to the other tasks. Moreover, Transformer exhibits a powerful generalization capacity because it has the property of dynamic feature weighting. Hence, there is a large potential of a uniform Transformer to learn multiple tasks simultaneously, i.e., multi-task learning (MTL). An MTL Transformer can combine knowledge from different tasks by sharing a uniform network. In this study, a general-purpose Transformer, which simultaneously processes the three tasks, is investigated for RS MTL. To build a Transformer capable of the three tasks, an MTL framework named RSCoTr is proposed. The framework uses a shared encoder to extract multi-scale features efficiently and three task-specific decoders to obtain different results. Moreover, a flexible training procedure named co-training is proposed. The MTL model is trained with multiple general data sets annotated for individual tasks. The co-training is as easy as training a specialized model for a single task. It can be developed into different learning strategies to meet various requirements. The proposed RSCoTr is trained jointly with various strategies on three challenging data sets of the three tasks. And the results demonstrate that the proposed MTL method achieves state-of-the-art performance in comparison with other competitive approaches. Code will be available at https://github.com/Li-Qingyun/RSCoTr.

In recent years, deep learning models have been widely used for hyperspectral image (HSI) classification and most of existing deep learning-based methods merely focused on high classification accuracy. However, in real applications, classification with low uncertainty matters as much as accurate classification. Unfortunately, existing methods fail to consider uncertainty. To tackle this challenge, for the first time, Bayesian deep learning (BDL) is investigated to analyze the model uncertainty for HSI classification. Specifically, first, at the feature extraction stage, an HSI classification framework based on BDL, which contains two Bayesian Gabor layers and a global pooling layer (i.e., BDL-G2), is proposed. In BDL-G2, parameters in Gabor layers are sampled from the Gaussian distribution. The proposed BDL-G2not only provides the uncertainty estimation, but also strengthens the structure characteristic (i.e., texture) of HSI. Second, to model the uncertainty at the final classification stage, BDL-G2is combined with a Bayesian fully-connected layer (i.e., BDL-G2-BFL), where the parameters’ distribution is adjusted adaptively. In the proposed BDL-G2-BFL, the uncertainty at feature extraction and classification stages are both captured, and a whole uncertainty estimation framework is established. Experimental results on the three public HSI datasets demonstrates the superiority in both accuracy and uncertainty. The proposed Bayesian deep learning-based methods pioneer a new direction and provide useful inspiration and experience for practical applications.

Recently, due to the powerful capability at modeling the long-range relationships, Transformer-based methods have been widely explored in many research areas including hyperspectral image (HSI) classification. However, because of lots of trainable parameters and the lack of inductive bias, it is difficult to train a Transformer-based HSI classifier, especially when the number of training samples is limited. To address this issue, in this study, spectral-spatial masked Transformer (SS-MTr) is explored for HSI classification, which uses a two-stage training strategy. In the first stage, SS-MTr pre-trains a vanilla Transformer via reconstruction from masked HSI inputs, which embeds the local inductive bias into the Transformer. In the second stage, the well pre-trained Transformer is cooperated with a fully connected layer and then fine-tuned for the HSI classification. Furthermore, in order to incorporate discriminative feature learning into the SS-MTr, three SS-MTr-based methods, including contrastive SS-MTr (C-SS-MTr), supervised SS-MTr (S-SS-MTr), and supervised contrastive SS-MTr (SC-SS-MTr) are proposed by adding extra branches for specific tasks in parallel with the existing reconstruction task. Specifically, the proposed C-SS-MTr adds a contrastive loss which brings instance discriminability. Besides, the proposed S-SS-MTr builds an extra classification branch for embracing inter-class discriminability and intra-class similarity. Moreover, the proposed SC-SS-MTr combines C-SS-MTr and S-SS-MTr for better generalization. The proposed SS-MTr, C-SS-MTr, S-SS-MTr, and SC-SS-MTr are tested on three popular hyperspectral datasets (i.e., Indian Pines, Pavia University, and Houston). The obtained results reveal that the proposed models achieve competitive results compared with the state-of-the-art HSI classification methods. Code is available at https://github.com/mengduanjinghua/SS-MTr.

Owing to its capability of building long-range dependencies and global context connections, Transformer has been used for hyperspectral image (HSI) classification. However, most of the existing Transformer-based HSI spatial–spectral classification methods consist of a convolutional neural network (CNN) and Transformer, which are used to extract the local and global information, respectively. In this study, to fully explore the potential of Transformer, a pure Transformer is investigated for HSI classification. First, a spatial Transformer (Spa-TR) is designed for HSI spatial classification, which learns the spatial features locally and globally by adopting the window partition and shifted window schemes. Especially, the self-attention computations are limited within the local windows and cross-windows. Second, to fully use the abundant spectral information in HSIs, a two-branch pure Transformer (i.e., Spa-Spe-TR) is proposed, which includes a spectral Transformer (Spe-TR) and a Spa-TR. The spectral sequence features learned by Spe-TR and the spatial features generated by Spa-TR are effectively fused with a branch fusion strategy, which explicitly and automatically measures the importance between the joint spatial–spectral features and improves the discriminability of the joint features. Experimental results on the two widely used HSI datasets (i.e., Pavia and Indian Pines) demonstrate the efficacy of the proposed methods in comparison with other state-of-the-art approaches.

The lack of training samples remains one of the major obstacles in applying convolutional neural networks (CNNs) to the hyperspectral image (HSI) classification. In this letter, the accurate classification of HSI with limited training samples is investigated. Due to the advantages of minimizing the distance between samples in the same class and maximizing the distance between samples in different classes, siamese CNN is used for HSI classification with limited training samples. After that, to improve the classification performance, data augmentation is investigated for siamese CNN-based HSI classification. Specifically, pair data augmentation based on CutMix is proposed to generate the training pairs of the same or different classes and the new generated training pairs are used to train siamese CNN. Traditional data augmentation methods simply generate new training samples. It is not proper when data augmentation methods keep the loss function and change the content of the input at the same time. Therefore, a soft-loss-based siamese CNN, which changes its loss according to the coupled replacement data augmentation, is proposed to further address the HSI classification with limited training samples. In the experimental part, on two widely used hyperspectral datasets, the influences of different training samples and window sizes are discussed, and the experimental results reveal that the proposed soft-augmentation-based siamese CNN provides competitive results with limited training samples compared with state-of-the-art methods.

Due to powerful feature extraction capability, convolutional neural networks (CNNs) have been widely used for hyperspectral image (HSI) classification. However, because of a large number of parameters that need to be trained, sufficient training samples are usually required for deep CNN-based methods. Unfortunately, limited training samples are a common issue in the remote sensing community. In this study, a dual graph convolutional network (DGCN) is proposed for the supervised classification of HSI with limited training samples. The first GCN fully extracts features existing in and among HSI samples, while the second GCN utilizes label distribution learning, and thus, it potentially reduces the number of required training samples. The two GCNs are integrated through several iterations to decrease interclass distances, which leads to a more accurate classification step. Moreover, a new idea entitled multiscale feature cutout is proposed as a regularization technique for HSI classification (DGCN-M). Different from the regularization methods (e.g., dropout and DropBlock), the proposed multiscale feature cutout could randomly mask out multiscale region sizes in a feature map, which further reduces the overfitting problem and yields consistent improvement. Experimental results on the four popular hyperspectral data sets (i.e., Salinas, Indian Pines, Pavia, and Houston) indicate that the proposed method obtains good classification performance compared to state-of-the-art methods, which shows the potential of GCN for HSI classification.

Recently, deep convolutional neural networks (CNNs) have achieved high classification accuracy of hyperspectral image (HSI). However, high accuracy is not the only goal of a good HSI classifier. In real-world applications, it is necessary to tell whether the classifier is certain about its classification result, which is critical for the safe usage. Unfortunately, most of existing models do not consider the issue. In this study, uncertainty is estimated and reduced to build a trustworthy HSI classifier. Firstly, since the output probabilities of softmax layer cannot represent the confidence scores, distance measurement scheme is used to measure the confidence scores. And then, a trustworthy HSI classifier, which reduces the predictive uncertainty in CNN (i.e., PU-CNN), is obtained by minimizing the distance to the correct centroid. Secondly, the fact that a training sample of HSI usually contains many pixel vectors that belong to different classes, which brings label uncertainty. Then, label uncertainty CNN (i.e., LU-CNN), which uses a classifier-consistent estimator to recover the multiple classes in each HSI sample, is proposed. LU-CNN computes loss over candidate label sets to find the optimal classes, which leads to a trustworthy HSI classifier. Finally, the combination of PU-CNN and LU-CNN (i.e., PL-CNN) is proposed to address predictive uncertainty and label uncertainty at the same time. Experimental results on the three popular hyperspectral datasets show that the proposed methods yield improvements in both accuracy and confidence. The proposed trustworthy classifier opens a new window for safe usage of HSI.

In recent years, deep convolutional neural networks (CNNs) have been widely used for hyperspectral image (HSI) classification. Besides, ensemble learning is a useful way to enhance the classification performance. Therefore, in this study, a new method titled Boosting-CNN is proposed for HSI classification, which fully explored the advantages of deep CNN and ensemble learning. Specifically, several deep CNNs are well-designed to classify HSI. The samples are misclassified in a CNN have more weighs in the following CNN through adaptive boosting. The final classification result is obtained by weighted voting of several CNNs. For HSI classification issue, the number of samples in different classes varies greatly, however, traditional classification methods cannot handle this issue well. In order to address imbalance training samples in HSI classification, soft class balanced loss is proposed to mitigate the influence of imbalance training samples. Experimental results on two popular hyperspectral datasets (i.e., Salinas and Pavia University) show that the proposed method obtain better classification accuracy compared to comparison methods.

In recent years, deep convolutional neural networks (CNNs) have been widely investigated for hyperspectral image (HSI) classification. The CNN-based HSI classifiers obtained good performance under the condition of sufficient training samples. In order to address the problem of limited training samples, in this letter, transfer learning is combined with CNN to address the issue of HSI classification. Pretrained models on large-scale data sets (e.g., ImageNet) can extract the general and discriminative features. Due to the fact that the extracted low-level and mid-level features can be reused for the HSI feature extraction, the CNN-based methods usually obtain good classification performance with insufficient training samples. The ImageNet data set has three channels, while the HSI data set contains hundreds of channels. Therefore, three channels of the HSI are randomly selected to formulate a transferring CNN. Then, several transferring CNNs are combined to establish an ensemble classification system with diversity. Moreover, an improved label smoothing technique is proposed to further improve the classification accuracy of the HSI. Experimental results on two popular hyperspectral data sets [i.e., Indian Pines and Kennedy Space Center (KSC)] show that the transferring CNN ensemble obtains good classification performance compared to the state-of-the-art methods.

Deep convolutional neural networks (CNNs) have shown their outstanding performance in the hyperspectral image (HSI) classification. The success of CNN-based HSI classification relies on the availability sufficient training samples. However, the collection of training samples is expensive and time consuming. Besides, there are many pretrained models on large-scale data sets, which extract the general and discriminative features. The proper reusage of low-level and midlevel representations will significantly improve the HSI classification accuracy. The large-scale ImageNet data set has three channels, but HSI contains hundreds of channels. Therefore, there are several difficulties to simply adapt the pretrained models for the classification of HSIs. In this article, heterogeneous transfer learning for HSI classification is proposed. First, a mapping layer is used to handle the issue of having different numbers of channels. Then, the model architectures and weights of the CNN trained on the ImageNet data sets are used to initialize the model and weights of the HSI classification network. Finally, a well-designed neural network is used to perform the HSI classification task. Furthermore, attention mechanism is used to adjust the feature maps due to the difference between the heterogeneous data sets. Moreover, controlled random sampling is used as another training sample selection method to test the effectiveness of the proposed methods. Experimental results on four popular hyperspectral data sets with two training sample selection strategies show that the transferred CNN obtains better classification accuracy than that of state-of-the-art methods. In addition, the idea of heterogeneous transfer learning may open a new window for further research.

Deep learning-based methods, especially deep convolutional neural networks (CNNs), have shown their effectiveness for hyperspectral image (HSI) classification. In previous deep CNN-based HSI classification methods, a cuboid is empirically determined as the input. The dimensionalities of the cuboid, including height and weight, are crucial to the final classification results. Unfortunately, these superparameters (i.e., the dimensionalities of input cube) are hand-crafted, which means the inputs of a classifier are not optimized according to the specific hyperspectral dataset. In this letter, spatial transformation network (STN) is explored to obtain the optimal input for CNN-based HSI classification for the first time. STN is used to translate, rotate, and scale the original input to obtain optimized input for the following CNN. Moreover, in order to mitigate the overfitting problem in CNN-based HSI classification, DropBlock is introduced as a regularization technique for HSI accurate classification. Compared with dropout, which is a popular regularization technique, DropBlock obtains better classification accuracy. The proposed methods are tested on two widely used hyperspectral data sets (i.e., Salinas and Kennedy Space Center). The obtained experimental results show that the proposed methods provide competitive results compared with state-of-the-art methods including deep CNN-based methods.

Light detection and ranging (LiDAR) is a useful data acquisition technique, which is widely used in a variety of practical applications. The classification of LiDAR-derived rasterized digital surface model (LiDAR-DSM) is a fundamental technique in LiDAR data processing. In recent years, deep learning methods, especially convolutional neural networks (CNNs), have shown their capability in remote sensing areas, including LiDAR data processing. Traditional deep models empirically use a fixed neighborhood system as input to the network. Therefore, the weight and height of the input rectangle may not be optimal. In order to modify such handcrafted setting, a spatial transformation network is used here to identify optimal inputs. The transformed inputs are fed into a well-designed CNN to obtain the final classification results. Furthermore, morphological profiles are combined with spatial transformation CNN to further improve the classification accuracy. The proposed frameworks are tested on two LiDAR-DSMs (i.e., the Recology and Houston data sets). The experimental results show that the proposed models provide competitive results compared to the state-of-the-art methods. Furthermore, the proposed optimal input identification approach can also be found beneficial for other remote sensing applications.

Hyperspectral image (HSI) classification is a core task in the remote sensing community, and recently, deep learning-based methods have shown their capability of accurate classification of HSIs. Among the deep learning-based methods, deep convolutional neural networks (CNNs) have been widely used for the HSI classification. In order to obtain a good classification performance, substantial efforts are required to design a proper deep learning architecture. Furthermore, the manually designed architecture may not fit a specific data set very well. In this paper, the idea of automatic CNN for the HSI classification is proposed for the first time. First, a number of operations, including convolution, pooling, identity, and batch normalization, are selected. Then, a gradient descent-based search algorithm is used to effectively find the optimal deep architecture that is evaluated on the validation data set. After that, the best CNN architecture is selected as the model for the HSI classification. Specifically, the automatic 1-D Auto-CNN and 3-D Auto-CNN are used as spectral and spectral-spatial HSI classifiers, respectively. Furthermore, the cutout is introduced as a regularization technique for the HSI spectral-spatial classification to further improve the classification accuracy. The experiments on four widely used hyperspectral data sets (i.e., Salinas, Pavia University, Kennedy Space Center, and Indiana Pines) show that the automatically designed data-dependent CNNs obtain competitive classification accuracy compared with the state-of-the-art methods. In addition, the automatic design of the deep learning architecture opens a new window for future research, showing the huge potential of using neural architectures' optimization capabilities for the accurate HSI classification.

In recent years, deep learning-based methods, especially convolutional neural networks (CNNs), have shown their capabilities in remote sensing data processing. The efficacy of light detection and ranging (LiDAR) has been already proven in a wide variety of research areas. Most of the existing methods do not extract the informative features from LiDAR-derived rasterized digital surface models (LiDAR-DSM) data in a deep manner. In order to utilize the advantages of deep models for the classification of LiDAR-derived features, deep CNN is proposed here to hierarchically extract the robust and discriminant features of the input data. Moreover, morphological profiles and multiattribute profiles (MAPs) are investigated to enrich the inputs of the CNN and further to improve the ultimate classification performance. Furthermore, a new activation function, sigmoid-weighted linear units (SiLUs), is introduced. The proposed frameworks are tested on two LiDAR-DSMs (i.e., Bayview Park and Houston data sets). The MAP-CNNs with SiLU outperform original CNNs by 6.62% and 6.88% in terms of overall accuracy on Bayview Park and Houston data sets, respectively, when the number of training samples of each class is 40.