Deep learning (DL) methodology contributes a lot to the development of hyperspectral image (HSI) analysis community. However, it also makes HSI analysis systems vulnerable to adversarial attacks. To this end, we propose a masked spatial-spectral autoencoder (MSSA) in this article under self-supervised learning theory, for enhancing the robustness of HSI analysis systems. First, a masked sequence attention learning (MSAL) module is conducted to promote the inherent robustness of HSI analysis systems along spectral channel. Then, we develop a graph convolutional network (GCN) with learnable graph structure to establish global pixel-wise combinations. In this way, the attack effect would be dispersed by all the related pixels among each combination, and a better defense performance is achievable in spatial aspect. Finally, to improve the defense transferability and address the problem of limited labeled samples, MSSA employs spectra reconstruction as a pretext task and fits the datasets in a self-supervised manner. Comprehensive experiments over three benchmarks verify the effectiveness of MSSA in comparison with the state-of-the-art hyperspectral classification methods and representative adversarial defense strategies.

Abstract Due to the powerful ability in capturing the global information, transformer has become an alternative architecture of CNNs for hyperspectral image classification. However, general transformer mainly considers the global spectral information while ignores the multiscale spatial information of the hyperspectral image. In this paper, we propose a multiscale spectral–spatial convolutional transformer (MultiFormer) for hyperspectral image classification. First, the developed method utilizes multiscale spatial patches as tokens to formulate the spatial transformer and generates multiscale spatial representation of each band in each pixel. Second, the spatial representation of all the bands in a given pixel are utilized as tokens to formulate the spectral transformer and generate the multiscale spectral–spatial representation of each pixel. Besides, a modified spectral–spatial CAF module is constructed in the MultiFormer to fuse cross‐layer spectral and spatial information. Therefore, the proposed MultiFormer can capture the multiscale spectral–spatial information and provide better performance than most of other architectures for hyperspectral image classification. Experiments are conducted over commonly used real‐world datasets and the comparison results show the superiority of the proposed method.

Convolutional neural networks (CNNs) have shown their potential ability to extract discriminative features for hyperspectral image classification. However, traditional deep learning methods using CNNs tend to overlook the influence of complex environmental factors. These factors contribute to an increase in intraclass variance and a decrease in interclass variance, making it considerably more challenging to extract meaningful features. To overcome this problem, this work develops a novel deep intrinsic decomposition with adversarial learning, namely AdverDecom, for hyperspectral image classification to mitigate the negative impact of environmental factors on classification performance. First, we develop a generative network for hyperspectral images (HyperNet) to extract the environment-related features and category-related features from the image. Then, a discriminative network is constructed to distinguish different environmental categories. Finally, an environment-category joint learning loss is developed for adversarial learning to make the deep model learn discriminative features. Experiments are conducted over four commonly used real-world datasets and the comparison results show the superiority of the proposed method. The implementation of the proposed method could be accessed athttps://github.com/shendu-sw/Adversarial_Learning_Intrinsic_Decompositionfor the sake of reproducibility.

The dissection of hyperspectral images into intrinsic components through hyperspectral intrinsic image decomposition (HIID) enhances the interpretability of hyperspectral data, providing a foundation for more accurate classification outcomes. However, the classification performance of HIID is constrained by the model’s representational ability. To address this limitation, this study rethinks hyperspectral intrinsic image decomposition for classification tasks by introducing deep feature embedding. The proposed framework, HyperDID, incorporates the Environmental Feature Module (EFM) and Categorical Feature Module (CFM) to extract intrinsic features. Additionally, a Feature Discrimination Module (FDM) is introduced to separate environment-related and category-related features. Experimental results across three commonly used datasets validate the effectiveness of HyperDID in improving hyperspectral image classification performance. This novel approach holds promise for advancing the capabilities of hyperspectral image analysis by leveraging deep feature embedding principles. The implementation of the proposed method could be accessed soon at https://github.com/shendu-sw/HyperDID for the sake of reproducibility.

Vision transformers (ViTs) have achieved great success in unmanned aerial vehicle (UAV) target detection tasks. However, little attention has been paid to the adversarial attack against ViT-based detectors, and the generated adversarial examples cannot take physical realizability and attack transferability into account at the same time. To overcome the limitation, we focus on transferable attacks toward ViT-based detectors in optical UAV-based remote sensing images and generate adversarial examples in the physical world. Concretely, we design unique perturbation patches deployed within and beyond the target object rather than requiring the patches to be aligned with image tokens. To narrow the gap between limited digital samples and complex physical scenarios, we conduct data augmentation on training images at global and local levels. In addition, we propose a novel transferable attack method named Jacobian matrix regularization (JMR), which consists of feature variance regularization (FVR) and attention weight regularization (AWR). Specifically, FVR calculates feature variances of different channels within specific layers and then sets the features as zeros for channels with top variances. AWR is achieved by masking the largest self-attention weights. We conduct extensive transferable experiments with typical detectors in both digital and physical UAV-based remote sensing scenarios. The results indicate that our method could achieve competitive transferability compared with state-of-the-art methods.

Full-state estimation with a limited number of sen-sors is a valuable and challenging task in monitoring and con-trolling complex physical systems. Supervised learning methods based on deep neural networks have shown excellent performance by learning the nonlinear mapping from sparse observations to global field. However, The neural network is a black box with weak explanation for physical processes, and the reconstruction performance is limited to the architecture and optimization of neural network. This paper aims to leverage the structure and laws inherent in data to reconstruct the global field by solving optimization problems instead of single network learning. We propose a unified global field reconstruction framework consisting of neural network prediction, proper orthogonal de-composition (POD), and linear optimization problem solving. The deep neural network is first trained to provide referenced global fields, which are combined with exact observations and the reference modes extracted by POD to establish a linear optimization problem. The objective of optimization problem is to superpose POD modes to satisfy the values of observations and referenced fields. The experiments conducted on fluid and thermal field reconstruction problems show that the proposed unified framework can significantly improve the reconstruction accuracy of neural networks and boost the performance of directly solving optimization problems without referenced fields.

Abstract Deep Neural Networks have been successfully applied in hyperspectral image classification. However, most of prior works adopt general deep architectures while ignore the intrinsic structure of the hyperspectral image, such as the physical noise generation. This would make these deep models unable to generate discriminative features and provide impressive classification performance. To leverage such intrinsic information, this work develops a novel deep learning framework with the noise inclined module and denoise framework for hyperspectral image classification. First, the spectral signature of hyperspectral image is modeled with the physical noise model to describe the high intra‐class variance of each class and great overlapping between different classes in the image. Then, a noise inclined module is developed to capture the physical noise within each object and a denoise framework is then followed to remove such noise from the object. Finally, the CNN with noise inclined module and the denoise framework is developed to obtain discriminative features and provides good classification performance of hyperspectral image. Experiments are conducted over two commonly used real‐world datasets and the experimental results show the effectiveness of the proposed method. The implementation of the proposed method and other compared methods could be accessed at https://github.com/shendu‐sw/noise‐physical‐framework .

The research on attack transferability is of great importance as it can guide how to conduct an adversarial attack without knowing any information about target models. However, it remains challenging for adversarial examples to maintain a good attack transferability performance, especially for the black-box attack implemented in the physical world. To enhance black-box transferability of physical attacks on object detectors, we present a novel adversarial learning method to produce adversarial patches by redistributing separable attention maps. Concretely, we first develop smoothed multilayer attention maps by introducing serial composite transformations, which could suppress model-specific noise on the one hand, and cover objects to be concealed at various resolutions on the other hand. Besides, our method resorts to a scalable mask to separate object attention from the background and adjust their distribution with a novel loss function. Extensive experiments show that our approach outperforms state-of-the-art methods in both the digital space and the physical world. Our code is available at https://github.com/zhangyu13a/transPhyAtt .

Recently, deep learning methods have demonstrated their potentials in extracting spectral information for hyperspectral images and have been widely applied in hyperspectral target detection (HTD). However, prior deep learning methods, represented by the convolutional neural networks, mainly focus on the local information and representation, which cannot well capture the long-range dependence. Besides, limited target references cannot meet the need of massive labelled samples for the training process. This work develops a triplet spectral-wise transformer-based target detector (TSTTD) to deal with these problems. First, this work explores a novel triplet spectral-wise transformer network for HTD task, and a data augmentation method is utilized to construct sufficient and balanced training samples for balanced learning. The proposed network shows advantages in learning local features from multiple adjacent bands and global features with long-range dependence. Second, for improving the separability between targets and backgrounds, a novel inter-category separation and intra-category aggregation (ISIA) loss function is proposed, which joints the hard-negative-mining triplet loss and the binary cross entropy loss. Third, experimental results on six data sets show that our proposed method is effective in leading to excellent detection performance when compared with other state-of-the-art methods.

Physical adversarial attacks in object detection have attracted increasing attention. However, most previous works focus on hiding the objects from the detector by generating an individual adversarial patch, which only covers the planar part of the vehicle’s surface and fails to attack the detector in physical scenarios for multi-view, long-distance and partially occluded objects. To bridge the gap between digital attacks and physical attacks, we exploit the full 3D vehicle surface to propose a robust Full-coverage Camouflage Attack (FCA) to fool detectors. Specifically, we first try rendering the nonplanar camouflage texture over the full vehicle surface. To mimic the real-world environment conditions, we then introduce a transformation function to transfer the rendered camouflaged vehicle into a photo-realistic scenario. Finally, we design an efficient loss function to optimize the camouflage texture. Experiments show that the full-coverage camouflage attack can not only outperform state-of-the-art methods under various test cases but also generalize to different environments, vehicles, and object detectors.

For the temperature field reconstruction (TFR), a complex image-to-image regression problem, the convolutional neural network (CNN) is a powerful surrogate model due to the convolutional layer's good image feature extraction ability. However, a lot of labeled data is needed to train CNN, and the common CNN can not quantify the aleatoric uncertainty caused by data noise. In actual engineering, the noiseless and labeled training data is hardly obtained for the TFR. To solve these two problems, this paper proposes a deep Monte Carlo quantile regression (Deep MC-QR) method for reconstructing the temperature field and quantifying aleatoric uncertainty caused by data noise. On the one hand, the Deep MC-QR method uses physical knowledge to guide the training of CNN. Thereby, the Deep MC-QR method can reconstruct an accurate TFR surrogate model without any labeled training data. On the other hand, the Deep MC-QR method constructs a quantile level image for each input in each training epoch. Then, the trained CNN model can quantify aleatoric uncertainty by quantile level image sampling during the prediction stage. Finally, the effectiveness of the proposed Deep MC-QR method is validated by many experiments, and the influence of data noise on TFR is analyzed.

Deep learning methods have played a more important role in hyperspectral image classification. However, general deep learning methods mainly take advantage of the samplewise information to formulate the training loss while ignoring the intrinsic data structure of each class. Due to the high spectral dimension and great redundancy between different spectral channels in the hyperspectral image, these former training losses usually cannot work so well for the deep representation of the image. To tackle this problem, this work develops a novel deep manifold embedding method (DMEM) for deep learning in hyperspectral image classification. First, each class in the image is modeled as a specific nonlinear manifold, and the geodesic distance is used to measure the correlation between the samples. Then, based on the hierarchical clustering, the manifold structure of the data can be captured and each nonlinear data manifold can be divided into several subclasses. Finally, considering the distribution of each subclass and the correlation between different subclasses under data manifold, DMEM is constructed as the novel training loss to incorporate the special classwise information in the training process and obtain discriminative representation for the hyperspectral image. Experiments over four real-world hyperspectral image datasets have demonstrated the effectiveness of the proposed method when compared with general sample-based losses and showed superiority when compared with state-of-the-art methods.

Nowadays, deep learning methods, especially the convolutional neural networks (CNNs), have shown impressive performance on extracting abstract and high-level features from the hyperspectral image. However, the general training process of CNNs mainly considers the pixelwise information or the samples' correlation to formulate the penalization while ignores the statistical properties especially the spectral variability of each class in the hyperspectral image. These sample-based penalizations would lead to the uncertainty of the training process due to the imbalanced and limited number of training samples. To overcome this problem, this article characterizes each class from the hyperspectral image as a statistical distribution and further develops a novel statistical loss with the distributions, not directly with samples for deep learning. Based on the Fisher discrimination criterion, the loss penalizes the sample variance of each class distribution to decrease the intraclass variance of the training samples. Moreover, an additional diversity-promoting condition is added to enlarge the interclass variance between different class distributions, and this could better discriminate samples from different classes in the hyperspectral image. Finally, the statistical estimation form of the statistical loss is developed with the training samples through multivariant statistical analysis. Experiments over the real-world hyperspectral images show the effectiveness of the developed statistical loss for deep learning.

A practical hyperspectral target characterization task estimates a target signature from imprecisely labeled training data. The imprecisions arise from the characteristics of the real-world tasks. First, accurate pixel-level labels on training data are often unavailable. Second, the subpixel targets and occluded targets cause the training samples to contain mixed data and multiple target types. To address these imprecisions, this paper proposes a new hyperspectral target characterization method to produce diverse multiple hyperspectral target signatures under a multiple instance learning (MIL) framework. The proposed method uses only bag-level training samples and labels, which solves the problems arising from the mixed data and lack of pixel-level labels. Moreover, by formulating a multiple characterization MIL and including a diversity-promoting term, the proposed method can learn a set of diverse target signatures, which solves the problems arising from multiple target types in training samples. The experiments on hyperspectral target detections using the learned multiple target signatures over synthetic and real-world data show the effectiveness of the proposed method.

This work develops a novel end-to-end deep unsupervised learning method based on convolutional neural network (CNN) with pseudo-classes for remote sensing scene representation. First, we introduce center points as the centers of the pseudo classes and the training samples can be allocated with pseudo labels based on the center points. Therefore, the CNN model, which is used to extract features from the scenes, can be trained supervised with the pseudo labels. Moreover, a pseudo-center loss is developed to decrease the variance between the samples and the corresponding pseudo center point. The pseudo-center loss is important since it can update both the center points with the training samples and the CNN model with the center points in the training process simultaneously. Finally, joint learning of the pseudo-center loss and the pseudo softmax loss which is formulated with the samples and the pseudo labels is developed for unsupervised remote sensing scene representation to obtain discriminative representations from the scenes. Experiments are conducted over two commonly used remote sensing scene datasets to validate the effectiveness of the proposed method and the experimental results show the superiority of the proposed method when compared with other state-of-the-art methods.

Recently, researchers have shown the powerful ability of deep methods with multilayers to extract high-level features and to obtain better performance for hyperspectral image classification. However, a common problem of traditional deep models is that the learned deep models might be suboptimal because of the limited number of training samples, especially for the image with large intraclass variance and low interclass variance. In this paper, novel convolutional neural networks (CNNs) with multiscale convolution (MS-CNNs) are proposed to address this problem by extracting deep multiscale features from the hyperspectral image. Moreover, deep metrics usually accompany with MS-CNNs to improve the representational ability for the hyperspectral image. However, the usual metric learning would make the metric parameters in the learned model tend to behave similarly. This similarity leads to obvious model's redundancy and, thus, shows negative effects on the description ability of the deep metrics. Traditionally, determinantal point process (DPP) priors, which encourage the learned factors to repulse from one another, can be imposed over these factors to diversify them. Taking advantage of both the MS-CNNs and DPP-based diversity-promoting deep metrics, this paper develops a CNN with multiscale convolution and diversified metric to obtain discriminative features for hyperspectral image classification. Experiments are conducted over four real-world hyperspectral image data sets to show the effectiveness and applicability of the proposed method. Experimental results show that our method is better than original deep models and can produce comparable or even better classification performance in different hyperspectral image data sets with respect to spectral and spectral-spatial features.

Recently, deep models have shown powerful ability for remote sensing scene representation. However, the training process of these deep methods requires large amount of labelled samples while usual remote sensing image datasets cannot provide enough training samples. Therefore, the learned model is usually suboptimal. To solve the problem, this work focuses on obtaining multiple choices by training multiple models simultaneously, and then the human oracle can choose a proper one from these choices. However, training several models separately usually makes the obtained results similar. This paper tries to diversify the obtained choices by encouraging the obtained choices to repulse from each other. Experiments are conducted on Ucmerced Land Use dataset to validate the effectiveness of the proposed method to provide multiple diversified choices.

Unsupervised learning of a convolutional neural network (CNN) is a feasible method to represent and classify remote sensing images, where labeling the observed data to prepare training samples is a highly expensive and time-consuming task. In this letter, we propose an unsupervised convolutional feature fusion network to formulate an easy-to-train but effective CNN representation of remote sensing images. The efficiency and effectiveness are derived from the following two aspects. First, the proposed method trains a deep CNN through unsupervised learning of each CNN layer in a greedy layer-wise manner, which makes the training relatively easy and efficient. Second, the feature fusion strategy in the proposed network can effectively use both the information from individual layers and the important interactions between different layers. As a result, the proposed network requires only several layers to obtain comparable or even better results than very deep networks. The experiments on unsupervised deep representations and the classification of remote sensing images demonstrate the efficiency and effectiveness of the proposed method.

Deep models with multiple layers have demonstrated their potential in learning abstract and invariant features for better representation and classification of remote sensing images. Moreover, metric learning (ML) is usually introduced into the deep models to further increase the discrimination of deep representations. However, the usual deep ML methods treat the training samples in each training batch in the stochastic gradient descent-based learning procedure independently, and thus, they neglect the important contextual (structural) information in the training samples. In this paper, we first introduce deep structural ML (DSML) into the literature of remote sensing scene classification and specifically capture and use the structural information during the training on the remote sensing images. Further analysis demonstrates that DSML usually makes many learned metric parameters similar. This similarity leads to obvious model redundancy and thus decreases the representational ability of the model. To address this problem, this paper proposes a new diversity-promoting DSML (D-DSML) method by regularizing the learning procedure by a diversity-promoting prior over the parameter factors. The proposed D-DSML encourages the parameter factors to be uncorrelated, such that each factor can model unique information, and thus, the model's description ability and classification performance would be significantly improved. Experiments over six real-world remote sensing scene data sets demonstrate that the proposed method obtains much better results than those obtained by the original deep models and has comparable or even better performances when compared with state-of-the-art methods.

In this work, a diversified deep structural metric learning is proposed for remote sensing image classification. Firstly, a deep structural metric learning is introduced to take full advantage of structural information of training batches. Secondly, we impose a diversity regularization over the factors of deep structural metric learning to encourage them to be uncorrelated, such that each factor tends to model unique information during the training phase and all factors sums up to capture a larger proportion of information. The diversified model could benefit the classification of remote sensing images. Experiments are conducted on two real-world remote sensing image datasets to evaluate the effectiveness and wide applicability of the proposed approach. The results show that our proposed method can obtain comparable or even better results on remote sensing image classification when compared with the recent results.

There are many attempts that utilize deep learning methods to solve the problem of classification in remote sensing images. Convolutional Neural Networks (CNN) have made very good performance for various visual tasks, and marked their important place in all deep learning models. However, for some classification tasks of remote sensing images, CNN could not demonstrate their full potential because of lacking large amounts of labeled training data. Some efforts have been made to combine CNN with unlabeled data to tackle the problem by performing unsupervised learning. In this work we propose the balanced data driven sparsity to help train CNN in an unsupervised way. The experiments over the real world remote sensing images demonstrate that the proposed method improves the performance of the recent methods.

In the literature of remote sensing, deep models with multiple layers have demonstrated their potentials in learning the abstract and invariant features for better representation and classification of hyperspectral images. The usual supervised deep models, such as convolutional neural networks, need a large number of labeled training samples to learn their model parameters. However, the real-world hyperspectral image classification task provides only a limited number of training samples. This paper adopts another popular deep model, i.e., deep belief networks (DBNs), to deal with this problem. The DBNs allow unsupervised pretraining over unlabeled samples at first and then a supervised fine-tuning over labeled samples. But the usual pretraining and fine-tuning method would make many hidden units in the learned DBNs tend to behave very similarly or perform as “dead” (never responding) or “potential over-tolerant” (always responding) latent factors. These results could negatively affect description ability and thus classification performance of DBNs. To further improve DBN's performance, this paper develops a new diversified DBN through regularizing pretraining and fine-tuning procedures by a diversity promoting prior over latent factors. Moreover, the regularized pretraining and fine-tuning can be efficiently implemented through usual recursive greedy and back-propagation learning framework. The experiments over real-world hyperspectral images demonstrated that the diversity promoting prior in both pretraining and fine-tuning procedure lead to the learned DBNs with more diverse latent factors, which directly make the diversified DBNs obtain much better results than original DBNs and comparable or even better performances compared with other recent hyperspectral image classification methods.

This work shows how to improve hyperspectral image classification through using both a deep representation and contextual information. To implement this objective, this work proposes a new Conditional Random Field (CRF) model (named DBN-CRF) with potentials defined over deep features produced by the Deep Belief Networks (DBNs). The newly formulated DBN-CRF model takes advantage of strength of the DBNs in learning a good representation and the ability of CRFs to model contextual (spatial) information in both observations and labels. Within a piecewise training framework, an efficient training method is proposed to train the whole DBN-CRF model end-to-end. This means that parameters in DBN and CRF can be jointly trained and thus the proposed method can fully use the strength of both DBN and CRF. Moreover, in the proposed training method, the end-to-end training can be implemented with a standard back-propagation algorithm, avoiding the repeated inference usually involved in CRF training and thus is computationally efficient. Experiments on real-world hyperspectral data show that our method outperforms the most recent approaches in hyperspectral image classification.

In this paper, a spectral-spatial classification method with Gaussian process was proposed for hyperspectral image classification. This method exploits the relationship among adjacent pixels and integrates it into spectral information to obtain spectral-spatial classification. In the proposed approach, the spatial information of a single pixel is weighted by the cosine similarity value between the adjacent pixels in the neighborhood. Experiments were conducted on the AVIRIS Indian Pines data set to evaluate the performance of the proposed approach. And the results demonstrated the effectiveness of the proposed methods to improve the classification performance by consideration of the spatial relationship between adjacent pixels in the hyperspectral image.