Multimodal sentiment analysis has emerged as a critical research area. However, existing methods face significant challenges: (1) Unimodal feature extraction techniques often fail to capture the topological structure within data, and do not effectively integrate local and global information, leading to information loss. (2) Traditional multimodal fusion methods, such as concatenation, addition, and multiplication, struggle to model modality differences and inter-modal correlations. In this paper, we propose a novel multi-task hypergraph-attention framework (MTHA) to improve feature discrimination and model performance. Experimental results demonstrate that MTHA outperforms most baseline models in both sentiment classification and regression.

Recent studies have shown that the deep domain adaptation (DA) technique has achieved remarkable results in cross-domain hyperspectral image (HSI) classification task. However, these DA methods assume that the source and target domains share the same classes, which may not hold true in real-world applications. Under open-set conditions, since the target domain may contain classes unseen in the source domain, direct domain alignment can lead to negative transfer phenomena. Moreover, the presence of multiple unknown classes in the target domain makes it difficult to learn more discriminative classification boundaries between known and unknown classes. To address these issues, we propose an open-set DA (OSDA) method for HSI classification based on weighted generative adversarial networks and dynamic thresholding (WGDT). First, we introduce a class anchor (CA) strategy to learn the metric space of known classes in the source domain. By calculating the similarity between the target-domain samples and the CA, we compute the reliability weights of the samples belonging to known classes. Then, based on these weights, we design an instance-level weighted-domain adversarial learning strategy to better align samples that are more likely to belong to known classes, avoiding negative transfer phenomena. Finally, we propose a dynamic thresholding method to learn the classification boundaries between known and unknown classes in the feature space and reject unknown class samples, thereby separating known class samples in the target domain. The experimental results on four cross-scene HSI classification tasks demonstrate that our proposed method outperforms some existing methods. The code is available athttps://github.com/Li-ZK/WGDT.

Most hyperspectral image (HSI) classification methods assume that all classes in the test set are present during training. However, in real-world applications, acquiring labeled training samples is challenging. As a result, it is difficult for the training dataset to cover all possible land cover types, leading to the generalized zero-shot learning (GZSL) problem. Recently, vision-language models (VLMs) have provided rich semantic priors for land cover classes, offering promising potential for GZSL. However, two fundamental gaps hinder their application to HSI classification: the task paradigm gap, arising from the difference between image-level VLMs and the pixel-level HSI classification task; and the knowledge gap, due to the inconsistency between VLM features and HSI spectral–spatial representations. To bridge both gaps, a novel framework leveraging VLM semantic priors for GZSL in HSI classification is proposed, primarily using pseudo-labeling technique to provide knowledge for unseen classes. Specifically, a pseudo-label generation and enhancement module enables a paradigm transition from image-level understanding to pixel-level classification by incorporating HSI’s spatial information. A pseudo-label correction module then refines noisy labels using spectral cues to address the knowledge gap. Finally, a global learning strategy integrates pseudo-label distillation, supervised learning, and feature regularization to classify seen classes while enabling generalization to unseen ones. Experiments on benchmark HSI datasets demonstrate the proposed method’s superiority in generalized zero-shot classification. This work highlights the potential of VLMs in advancing HSI classification in practical applications.

This work has delivered a novelty foundational model for hyperspectral remote sensing images. Current approaches for hyperspectral data interpretation often require specialized models that are specifically tailored to individual datasets or tasks. In some specific tasks, the availability of hyperspectral data is often limited, posing significant challenges to training due to data insufficiency. Furthermore, the diverse structure of hyperspectral data often complicates the transfer of knowledge from other available datasets, severely limiting cross-scenario capabilities. To bridge this gap, we introduce a highly adaptable foundational model with strong transferability, capable of processing all forms of hyperspectral data across diverse spectral bands and ranges. Compared to previous methods, our approach: 1) standardizes all types spectral vectors into a common token format, enabling a single model for multi-source hyperspectral data; 2) aligns spectral features across different ranges by embedding wavelength information into position encoding; 3) has been pre-trained on over 300 million spectral instances worldwide, ensuring broad generalization; 4) transfers learned knowledge effortlessly to downstream tasks and new datasets without architectural modifications or training from scratch. Experimental results demonstrate that, compared to other mainstream methods, our approach achieves state-of-the-art classification performance across various datasets with different spectral characteristics in both supervised and unsupervised learning settings, while also delivering impressive results in change detection tasks. The source code and the pretrained weights are available at https://github.com/kkweil/HyperSL.

In many military and civilian applications, synthetic aperture radar (SAR) image target detection plays a vital role. However, current methods for SAR target detection generally fail to balance speed and accuracy, thus making it impossible to deploy them to real-world engineering applications. In addition, strong scattering, multiscales, high density, complex background interference, and speckle noise make it remarkably challenging to extract effective target information and disentangle background noise from the target information, ultimately resulting in high missing and false alarm rates. To address these issues, a unified dynamic SAR target detection architecture (DAFDet) with asymptotic fusion enhancement and feature encoding decoupling is proposed in this article. First, a dynamic architecture is constructed by cascading two identical detectors and integrating a designed decision maker. This decision maker can automatically decide the inference route by calculating the difficulty score of an SAR image, which ensures efficient inference speed while achieving high accuracy. Second, an asymptotic fusion enhancement feature pyramid network (AFEFPN) is developed, which can avoid the loss and degradation of target information in multistage transmissions through direct interactions of nonadjacent levels, thereby enhancing the extraction of valid target information. Suppression of background noise is achieved by modeling the importance of different feature channels of the fused features. Finally, a task-oriented decoupled head (TODH) is proposed to boost the localization and classification abilities of the model in complex scenarios. It decouples feature encoding at the source, thus providing task-oriented feature context. Numerous experiments on four widely adopted datasets reveal that DAFDet obtains efficient inference speed and optimal detection accuracy, achieving new state-of-the-art target detection performance. The source code will be provided athttps://github.com/yangyahu-1994/DAFDet.

The target classification of synthetic aperture radar (SAR) is an important technique of SAR image processing. Recently, many deep convolutional neural network (CNN)-based methods have been proposed for SAR target classification. However, feature extraction abilities of these CNN-based methods are insufficient. On the other hand, attention-based methods (e.g., swin Transformer) have the advantage of capturing the local-global features of images. Moreover, there always exists inherently noise in SAR images influenced by the process of emitted pulses, which hinders the improvement of accuracy for SAR target classification. To solve the both problems, this study explores a swin Transformer with improved blind-spot network (STr-BS) to alleviate the bad influence caused by speckle noise in SAR image and enhance the classification result. Specifically, the denoising process in STr-BS designs an improved blind-spot network in unsupervised setting without requiring the clean SAR images as input. Then, the outputs of the improved blind-spot network are as the input of the swin Transformer for the subsequent local-global feature extraction. The proposed STr-BS is tested on two public datasets (MSTAR and OpenSARShip), and the experimental results demonstrate the effectiveness of the proposed methods in comparison to other state-of-the-art approaches.

Radar jamming recognition is a fundamental step for anti-jamming techniques. However, the recognition performance is impeded in insufficient samples situation. Ensemble learning provides an effective way to address this issue. In this paper, first, a novel ensemble learning-based radar Transformer (i.e., RadarTR-E) is proposed to improve recognition performance with insufficient samples. Specifically, it votes on the predictions among sub-recognizers to increase recognition accuracy, where RadarTR is employed to effectively capture long-range dependencies. Then, a dynamic label smoothing method (i.e., RadarTR-E-DLS) is further proposed to mitigate overfitting. In detail, dynamic soft labels are designed to prevent overconfidence towards certain radar jamming types. Therefore, the proposed RadarTR-E-DLS achieves better recognition accuracy. Compared with other advanced methods, the experimental results show the superior recognition performance of the proposed methods.

Transformer has been widely used in hyperspectral image (HSI) classification tasks because of its ability to capture long-range dependencies. However, most Transformer-based classification methods lack the extraction of local information or do not combine spatial and spectral information well, resulting in insufficient extraction of features. To address these issues, in this study, a dual-branch masked Transformer (Dual-MTr) model is proposed. Masked Transformer (MTr) is used to pretrain vision transformer (ViT) by reconstruction of both masked spatial image and spectral spectrum, which embeds the local bias by the process of recovering from localized patches to the global original input. Different tokenization methods are used for different types of input data. Patch embedding with overlapping regions is used for 2-D spatial data and group embedding is used for 1-D spectral data. Supervised learning has been added to the pretraining process to enhance strong discriminability. Then, the dual-branch structure is proposed to combine the spatial and spectral features. To strengthen the connection between the two branches better, Kullback-Leibler (KL) divergence is used to measure the differences between the classification results of the two branches, and the loss resulting from the computed differences is incorporated into the training process. Experimental results from two hyperspectral datasets demonstrate the effectiveness of the proposed method compared to other methods.

Recently, both large language models and large vision models (LVMs) have gained significant attention. Trained on large-scale datasets, these large models have showcased remarkable capabilities across various research domains. To enhance the accuracy of remote sensing (RS) scene classification, LVM-based methods are explored in this letter. Due to the differences between RS images and natural images, simply transferring LVMs to RS tasks is impractical. Therefore, we conducted research on relevant techniques and appended learnable prompt tokens to the input tokens while freezing the backbone weights, reducing the parameter scale and making the LVM weights easier to harness and to transfer. In consideration of latent catastrophic forgetting issues induced by ordinary finetuning techniques and the inherent complexity and redundancy of RS images, we introduced soft adaption mechanisms between backbone layers based on prompt tuning technique and implemented the first LVM tuning method, namely, the Large Vision Model Soft Adaption for RS scene classification (LVM-StARS)-Deep and the LVM-StARS-Shallow to make LVMs more suitable for RS scene classification tasks. The proposed methods are evaluated on two popular RS scene classification datasets, and the experimental results indicate that the proposed method outperforms other state-of-the-art methods. The experimental results demonstrate that our proposed method enhances overall accuracy (OA) by 1.71%–3.94%, while updating only 0.1%–0.5% of the parameters compared to full finetuning. Furthermore, our method outperforms the existing methods.

With the increasing availability and openness of remote sensing (RS) data collected from diverse sensors, there has been a growing interest in multimodal RS data classification. Nowadays, in the area of deep learning, there is a paradigm shift with the rise of foundation models, which are trained on large-scale datasets and are adaptable to a wide range of downstream tasks. In this study, the potential and effectiveness of foundation models for multimodal RS data classification is investigated. The training datasets of foundation models and multimodal RS datasets are quite different, and therefore, it is difficult to use a pretrained foundation model for multimodal RS data classification directly. To mitigate this difficulty, this article proposes a foundation model adaptation (FMA) framework for multimodal RS data classification without fine-tuning the parameters. Specifically, two learnable modules, i.e., cross-spatial interaction module and cross-channel interaction module, are proposed to add to the foundation model for extracting multimodal-specific representations. The cross-spatial and cross-channel interaction modules extract the characteristics of unimodal features along the spatial dimension and channel dimension, respectively. To effectively tackle the disparities among various RS modalities, an alignment approach (FMA2) is further explored based on the FMA. The FMA2 describes dependencies between different modalities by establishing a coupling score function, which can further enhance classification performance. To demonstrate the effectiveness and superiority of the FMA framework, comprehensive experiments are conducted on three multimodal RS datasets, showing improvement over the advanced multimodal RS data classification image methods.

Recently, deep learning models have dominated hyperspectral image (HSI) classification. Nowadays, deep learning is undergoing a paradigm shift with the rise of transformer-based foundation models. In this study, the potential of transformer-based foundation models, including the vision foundation model (VFM) and language foundation model (LFM), for HSI classification are investigated. First, to improve the performance of traditional HSI classification tasks, a spectral-spatial VFM-based transformer (SS-VFMT) is proposed, which inserts spectral-spatial information into the pretrained foundation transformer. Specifically, a given pretrained transformer receives HSI patch tokens for long-range feature extraction benefiting from the prelearned weights. Meanwhile, two enhancement modules, i.e., spatial and spectral enhancement modules (SpaEMs$\backslash $SpeEMs), utilize spectral and spatial information for steering the behavior of the transformer. Besides, an additional patch relationship distillation strategy is designed for SS-VFMT to exploit the pretrained knowledge better, leading to the proposed SS-VFMT-D. Second, based on SS-VFMT, to address a new HSI classification task, i.e., generalized zero-shot classification, a spectral-spatial vision-language-based transformer (SS-VLFMT) is proposed. This task is to recognize novel classes not seen during training, which is more meaningful as the real world is usually open. The SS-VLFMT leverages SS-VFMT to extract spectral-spatial features and corresponding hash codes while integrating a pretrained language model to extract text features from class names. Experimental results on HSI datasets reveal that the proposed methods are competitive compared to the state-of-the-art methods. Moreover, the foundation model-based methods open a new window for HSI classification tasks, especially for HSI zero-shot classification.

Several fundamental remote sensing (RS) image processing tasks, including classification, segmentation, and detection, have been set to serve for manifold applications. In the RS community, the individual tasks have been studied separately for many years. However, the specialized models were only capable of a single task. They lacked the adaptability for generalizing to the other tasks. Moreover, Transformer exhibits a powerful generalization capacity because it has the property of dynamic feature weighting. Hence, there is a large potential of a uniform Transformer to learn multiple tasks simultaneously, i.e., multi-task learning (MTL). An MTL Transformer can combine knowledge from different tasks by sharing a uniform network. In this study, a general-purpose Transformer, which simultaneously processes the three tasks, is investigated for RS MTL. To build a Transformer capable of the three tasks, an MTL framework named RSCoTr is proposed. The framework uses a shared encoder to extract multi-scale features efficiently and three task-specific decoders to obtain different results. Moreover, a flexible training procedure named co-training is proposed. The MTL model is trained with multiple general data sets annotated for individual tasks. The co-training is as easy as training a specialized model for a single task. It can be developed into different learning strategies to meet various requirements. The proposed RSCoTr is trained jointly with various strategies on three challenging data sets of the three tasks. And the results demonstrate that the proposed MTL method achieves state-of-the-art performance in comparison with other competitive approaches. Code will be available at https://github.com/Li-Qingyun/RSCoTr.

Spectral reconstruction (SR) techniques can generate hyperspectral images (HSIs) from multispectral images (MSIs) with the same spatial resolution, thus alleviating the problem of limited availability and low spatial resolution of satellite HSIs. However, in scenarios where both HSIs and MSIs can be acquired simultaneously, spectral mapping relationship (SMR) among real images may not align with the sensor’s spectral response function (SRF), due to factors such as sensor noise and calibration errors. This mismatch can result in discrepancies in reflectivity between the reconstructed HSIs and the real HSIs. To solve the above problems, this article proposes a semi-supervised transfer learning SR (SSTSR) model based on gradient direction constraints. Through semi-supervised learning, SSTSR acquires precise SMRs in overlapping regions and extracts spectral trend information of ground objects from historical models in nonoverlapping regions. Experiments on two datasets demonstrate that the reconstructed HSIs closely resemble real HSIs, leading to impressive classification performance when employing a real HSI classifier.

Recently, metric-based few-shot learning (FSL) methods have achieved good performance in hyperspectral image (HSI) classification. However, existing methods suffer from two problems: over-reliance on image modality information leads to inaccurate prototype representation, where a prototype refers to the centroid of each class in the dataset, and the impact of redundant and noisy pixels on model discriminability is rarely considered. These problems result in insufficient discriminability of the model for the target domain. To address the above issues, we propose a cross-domain few-shot HSI classification framework with cross-modal alignment and supervised contrastive learning (CDFS-CASCL). It is well known that human visual learning greatly benefits from the input of various modal information such as vision, language and video. Inspired by the way humans abstract image class concepts in language form and understand the essence of classes, we perform cross-modal alignment (CA) between similar image and text prototypes, and use abstract text semantics to guide the model to learn semantic related features with good generalization ability in images, so as to improve the accuracy of image prototypes representation of the prototypes. In addition, through supervised contrastive learning (SCL) based on neighborhood pixel mask in the target domain, the enhanced sample features belonging to the same class are closer, while the enhanced sample features belonging to different classes are pulled further, enabling the model to learn mask-robust discriminative feature representations, suppressing the negative impact of redundant and noisy pixels, and improving the model’s discriminability. The experimental results demonstrate the superiority of the proposed CDFS-CASCL. The code is available at https://github.com/Li-ZK/CDFS-CASCL-2024.

Existing oriented object detection in aerial images has progressed a lot in recent years and achieved a favorable success. However, high-precision oriented object detection in aerial images remains a challenging task. Some recent works have adopted the classification-based method to predict the angle in order to address boundary problem in angle. However, we have found that these works often neglect the sensitivity of objects with different aspect ratios to angle. At the same time, it is worth exploring a suitable way to improve the emerging transformer-based approaches in order to adapt them to oriented object detection. In this paper, we propose an Aspect Ratio Sensitive DEtection TRansformer, termed ARS-DETR, for oriented object detection in aerial images. Specifically, a new angle classification method, called Aspect Ratio aware Circle Smooth Label (AR-CSL), is proposed to smooth the angle label in a more reasonable way and discard the hyperparameter that introduced by previous work (e.g. CSL). Then, a rotated deformable attention module is designed to rotate the sampling points with the corresponding angles and eliminate the misalignment between region features and sampling points. Moreover, a dynamic weight coefficient according to the aspect ratio is adopted to calculate the angle loss. Comprehensive experiments on several challenging datasets demonstrate that our method achieves a competitive performance in the high-precision oriented object detection task.

In recent years, deep learning models have been widely used for hyperspectral image (HSI) classification and most of existing deep learning-based methods merely focused on high classification accuracy. However, in real applications, classification with low uncertainty matters as much as accurate classification. Unfortunately, existing methods fail to consider uncertainty. To tackle this challenge, for the first time, Bayesian deep learning (BDL) is investigated to analyze the model uncertainty for HSI classification. Specifically, first, at the feature extraction stage, an HSI classification framework based on BDL, which contains two Bayesian Gabor layers and a global pooling layer (i.e., BDL-G2), is proposed. In BDL-G2, parameters in Gabor layers are sampled from the Gaussian distribution. The proposed BDL-G2not only provides the uncertainty estimation, but also strengthens the structure characteristic (i.e., texture) of HSI. Second, to model the uncertainty at the final classification stage, BDL-G2is combined with a Bayesian fully-connected layer (i.e., BDL-G2-BFL), where the parameters’ distribution is adjusted adaptively. In the proposed BDL-G2-BFL, the uncertainty at feature extraction and classification stages are both captured, and a whole uncertainty estimation framework is established. Experimental results on the three public HSI datasets demonstrates the superiority in both accuracy and uncertainty. The proposed Bayesian deep learning-based methods pioneer a new direction and provide useful inspiration and experience for practical applications.

Recently, due to the powerful capability at modeling the long-range relationships, Transformer-based methods have been widely explored in many research areas including hyperspectral image (HSI) classification. However, because of lots of trainable parameters and the lack of inductive bias, it is difficult to train a Transformer-based HSI classifier, especially when the number of training samples is limited. To address this issue, in this study, spectral-spatial masked Transformer (SS-MTr) is explored for HSI classification, which uses a two-stage training strategy. In the first stage, SS-MTr pre-trains a vanilla Transformer via reconstruction from masked HSI inputs, which embeds the local inductive bias into the Transformer. In the second stage, the well pre-trained Transformer is cooperated with a fully connected layer and then fine-tuned for the HSI classification. Furthermore, in order to incorporate discriminative feature learning into the SS-MTr, three SS-MTr-based methods, including contrastive SS-MTr (C-SS-MTr), supervised SS-MTr (S-SS-MTr), and supervised contrastive SS-MTr (SC-SS-MTr) are proposed by adding extra branches for specific tasks in parallel with the existing reconstruction task. Specifically, the proposed C-SS-MTr adds a contrastive loss which brings instance discriminability. Besides, the proposed S-SS-MTr builds an extra classification branch for embracing inter-class discriminability and intra-class similarity. Moreover, the proposed SC-SS-MTr combines C-SS-MTr and S-SS-MTr for better generalization. The proposed SS-MTr, C-SS-MTr, S-SS-MTr, and SC-SS-MTr are tested on three popular hyperspectral datasets (i.e., Indian Pines, Pavia University, and Houston). The obtained results reveal that the proposed models achieve competitive results compared with the state-of-the-art HSI classification methods. Code is available at https://github.com/mengduanjinghua/SS-MTr.

Recently, few-shot learning (FSL) has been introduced for hyperspectral image (HSI) classification with few labeled samples. However, existing FSL-based HSI classification methods mainly focus on the meta-knowledge transfer between HSIs. Compared with HSIs, natural images have sufficient annotated data. To utilize natural images (base class data) to achieve accurate classification of HSIs (novel class data), we propose a novel few-shot classification framework with SSL (FSCF-SSL) for HSIs in this article. The orientation of objects in natural images is relatively unitary, whereas the objects of image patches for each pixel in HSIs have diverse orientations in the spatial domain. To make better use of base classes, we design an SSL with geometric transformations (SSLGTs), which sets rotation labels as supervision to extract low-level features that can better represent diverse orientations, and then conduct SSLGT and FSL on base classes to learn transferable spatial meta-knowledge. Next, a spectral-spatial feature extraction network is carefully designed to better utilize the spatial and spectral information of HSIs, where the weights of the first seven layers of the spatial part are initialized by the weights of the corresponding layers trained on base classes. Finally, to fully explore the few annotated data from novel classes, we design an SSL with contrastive learning (SSLCL) that can mine the category-invariant features contained in the novel class data itself, and then perform SSLCL and FSL on novel classes to learn more discriminative individual knowledge. Experimental results on four HSI datasets show that FSCF-SSL offers a significant improvement over state-of-the-art methods. The code is available athttps://github.com/Li-ZK/FSCF-SSL-2023.

Owing to its capability of building long-range dependencies and global context connections, Transformer has been used for hyperspectral image (HSI) classification. However, most of the existing Transformer-based HSI spatial–spectral classification methods consist of a convolutional neural network (CNN) and Transformer, which are used to extract the local and global information, respectively. In this study, to fully explore the potential of Transformer, a pure Transformer is investigated for HSI classification. First, a spatial Transformer (Spa-TR) is designed for HSI spatial classification, which learns the spatial features locally and globally by adopting the window partition and shifted window schemes. Especially, the self-attention computations are limited within the local windows and cross-windows. Second, to fully use the abundant spectral information in HSIs, a two-branch pure Transformer (i.e., Spa-Spe-TR) is proposed, which includes a spectral Transformer (Spe-TR) and a Spa-TR. The spectral sequence features learned by Spe-TR and the spatial features generated by Spa-TR are effectively fused with a branch fusion strategy, which explicitly and automatically measures the importance between the joint spatial–spectral features and improves the discriminability of the joint features. Experimental results on the two widely used HSI datasets (i.e., Pavia and Indian Pines) demonstrate the efficacy of the proposed methods in comparison with other state-of-the-art approaches.

Recently, many deep convolutional neural network (DCNN)-based methods have been proposed for remote sensing (RS) image scene classification (SC). In general, DCNNs obtain good generalization capabilities under the condition of correct labels. Unfortunately, the given samples are sometimes mislabeled. In this letter, the classification of RS images with noisy labels is investigated. First, complementary learning (CL), which learns from complementary labels rather than the original labels, is introduced for RS image classification with noisy labels. CL can decrease the probability of learning from incorrect information, and therefore, it is robust to noisy labels. Then, soft CL, which randomly disturbs the complementary labels of the training samples, is proposed to prevent the overfitting issue in training a DCNN. Moreover, an RS image scene classification framework combining ordinary learning (OL) and CL (RS-COCL) is proposed, which uses CL to obtain a good model and OL to fine-tune the deep model. Additionally, noisy labels filtering is used in RS-COCL (RS-COCL-NLF) to detected and corrected noisy samples. At last, soft CL is used in RS-COCL-NLF to obtain better classification performance. The proposed methods are tested on two widely used datasets (i.e., Northwestern Polytechnical University (NWPU)-RESISC45 and PatternNet) and the obtained results show that the proposed methods provide competitive classification accuracy compared to the state-of-the-art methods.

The lack of training samples remains one of the major obstacles in applying convolutional neural networks (CNNs) to the hyperspectral image (HSI) classification. In this letter, the accurate classification of HSI with limited training samples is investigated. Due to the advantages of minimizing the distance between samples in the same class and maximizing the distance between samples in different classes, siamese CNN is used for HSI classification with limited training samples. After that, to improve the classification performance, data augmentation is investigated for siamese CNN-based HSI classification. Specifically, pair data augmentation based on CutMix is proposed to generate the training pairs of the same or different classes and the new generated training pairs are used to train siamese CNN. Traditional data augmentation methods simply generate new training samples. It is not proper when data augmentation methods keep the loss function and change the content of the input at the same time. Therefore, a soft-loss-based siamese CNN, which changes its loss according to the coupled replacement data augmentation, is proposed to further address the HSI classification with limited training samples. In the experimental part, on two widely used hyperspectral datasets, the influences of different training samples and window sizes are discussed, and the experimental results reveal that the proposed soft-augmentation-based siamese CNN provides competitive results with limited training samples compared with state-of-the-art methods.

Deep learning has achieved great success in hyperspectral image (HSI) classification. However, its success relies on the availability of sufficient training samples. Unfortunately, the collection of training samples is expensive, time-consuming, and even impossible in some cases. Natural image datasets that are different from HSI, such as Image Net and mini-ImageNet, have abundant texture and structure information. Effective knowledge transfer between two heterogeneous datasets can significantly improve the accuracy of HSI classification. In this letter, heterogeneous few-shot learning (HFSL) for HSI classification is proposed with only a few labeled samples per class. First, few-shot learning is performed on the mini-ImageNet datasets to learn the transferable knowledge. Then, to make full use of the spatial and spectral information, a spectral–spatial fusion network is devised. Spectral information is obtained by the residual network with pure 1-D operators. Spatial information is extracted by a convolution network with pure 2-D operators, and the weights of the spatial network are initialized by the weights of the model trained on the mini-ImageNet datasets. Finally, few-shot learning is fine-tuned on HSI to extract discriminative spectral–spatial features and individual knowledge, which can improve the classification performance of the new classification task. Experiments conducted on two public HSI datasets demonstrate that the HFSL outperforms the existing few-shot learning methods and supervised learning methods for HSI classification with only a few labeled samples. Our source code is available athttps://github.com/Li-ZK/HFSL.

Cross-domain hyperspectral image classification is one of the major challenges in remote sensing, especially for target domain data without labels. Recently, deep learning approaches have demonstrated effectiveness in domain adaptation. However, most of them leverage unlabeled target data only from a statistical perspective but neglect the analysis at the instance level. For better statistical alignment, existing approaches employ the entire unevaluated target data in an unsupervised manner, which may introduce noise and limit the discriminability of the neural networks. In this article, we propose confident learning-based domain adaptation (CLDA) to address the problem from a new perspective of data manipulation. To this end, a novel framework is presented to combine domain adaptation with confident learning (CL), where the former reduces the interdomain discrepancy and generates pseudo-labels for the target instances, from which the latter selects high-confidence target samples. Specifically, the confident learning part evaluates the confidence of each pseudo-labeled target sample based on the assigned labels and the predicted probabilities. Then, high-confidence target samples are selected as training data to increase the discriminative capacity of the neural networks. In addition, the domain adaptation part and the confident learning part are trained alternately to progressively increase the proportion of high-confidence labels in the target domain, thus further improving the accuracy of classification. Experimental results on four datasets demonstrate that the proposed CLDA method outperforms the state-of-the-art domain adaptation approaches. Our source code is available athttps://github.com/Li-ZK/CLDA-2022.

Due to powerful feature extraction capability, convolutional neural networks (CNNs) have been widely used for hyperspectral image (HSI) classification. However, because of a large number of parameters that need to be trained, sufficient training samples are usually required for deep CNN-based methods. Unfortunately, limited training samples are a common issue in the remote sensing community. In this study, a dual graph convolutional network (DGCN) is proposed for the supervised classification of HSI with limited training samples. The first GCN fully extracts features existing in and among HSI samples, while the second GCN utilizes label distribution learning, and thus, it potentially reduces the number of required training samples. The two GCNs are integrated through several iterations to decrease interclass distances, which leads to a more accurate classification step. Moreover, a new idea entitled multiscale feature cutout is proposed as a regularization technique for HSI classification (DGCN-M). Different from the regularization methods (e.g., dropout and DropBlock), the proposed multiscale feature cutout could randomly mask out multiscale region sizes in a feature map, which further reduces the overfitting problem and yields consistent improvement. Experimental results on the four popular hyperspectral data sets (i.e., Salinas, Indian Pines, Pavia, and Houston) indicate that the proposed method obtains good classification performance compared to state-of-the-art methods, which shows the potential of GCN for HSI classification.

Recently, deep convolutional neural networks (CNNs) have achieved high classification accuracy of hyperspectral image (HSI). However, high accuracy is not the only goal of a good HSI classifier. In real-world applications, it is necessary to tell whether the classifier is certain about its classification result, which is critical for the safe usage. Unfortunately, most of existing models do not consider the issue. In this study, uncertainty is estimated and reduced to build a trustworthy HSI classifier. Firstly, since the output probabilities of softmax layer cannot represent the confidence scores, distance measurement scheme is used to measure the confidence scores. And then, a trustworthy HSI classifier, which reduces the predictive uncertainty in CNN (i.e., PU-CNN), is obtained by minimizing the distance to the correct centroid. Secondly, the fact that a training sample of HSI usually contains many pixel vectors that belong to different classes, which brings label uncertainty. Then, label uncertainty CNN (i.e., LU-CNN), which uses a classifier-consistent estimator to recover the multiple classes in each HSI sample, is proposed. LU-CNN computes loss over candidate label sets to find the optimal classes, which leads to a trustworthy HSI classifier. Finally, the combination of PU-CNN and LU-CNN (i.e., PL-CNN) is proposed to address predictive uncertainty and label uncertainty at the same time. Experimental results on the three popular hyperspectral datasets show that the proposed methods yield improvements in both accuracy and confidence. The proposed trustworthy classifier opens a new window for safe usage of HSI.

One of the challenges in hyperspectral image (HSI) classification is that there are limited labeled samples to train a classifier for very high-dimensional data. In practical applications, we often encounter an HSI domain (called target domain) with very few labeled data, while another HSI domain (called source domain) may have enough labeled data. Classes between the two domains may not be the same. This article attempts to use source class data to help classify the target classes, including the same and new unseen classes. To address this classification paradigm, a meta-learning paradigm for few-shot learning (FSL) is usually adopted. However, existing FSL methods do not account for domain shift between source and target domain. To solve the FSL problem under domain shift, a novel deep cross-domain few-shot learning (DCFSL) method is proposed. For the first time, DCFSL tackles FSL and domain adaptation issues in a unified framework. Specifically, a conditional adversarial domain adaptation strategy is utilized to overcome domain shift, which can achieve domain distribution alignment. In addition, FSL is executed in source and target classes at the same time, which can not only discover transferable knowledge in the source classes but also learn a discriminative embedding model to the target classes. Experiments conducted on four public HSI data sets demonstrate that DCFSL outperforms the existing FSL methods and deep learning methods for HSI classification. Our source code is available athttps://github.com/Li-ZK/DCFSL-2021.

Accurate classification of remote sensing (RS) images is perennial topic of interest in the RS community. Recently, transfer learning, especially for fine-tuning pre-trained convolutional neural networks (CNNs), has been proposed as a feasible strategy for RS scene classification. However, because the target domain (i.e., the RS images) and the source domain (e.g., ImageNet) are quite different, simply using the model pre-trained on an ImageNet dataset presents some difficulties. The RS images and the pre-trained models need to be properly adjusted to build a better classification system. In this study, an adaptive learning strategy for transferring a CNN-based model is proposed. First, an adaptive transform is used to adjust the original size of the RS image to a certain size, which is tailored to the input of the subsequent pre-trained model. Then, an adaptive transferring model is proposed to automatically learn what knowledge from the pre-trained model should be transferred to the RS scene classification model. Finally, in combination with a label smoothing approach, adaptive label is presented to generate soft labels based on the statistics of the classification model predictions for each category, which is beneficial for learning the relationships between the target and non-target categories of scenes. In general, the proposed methods adaptively manage the input, model, and label simultaneously, which leads to better classification performance for RS scene classification. The proposed methods are tested on three widely-used data sets and the obtained results show that the proposed methods provide competitive classification accuracy compared to the state-of-the-art methods.

In recent years, deep convolutional neural networks (CNNs) have been widely used for hyperspectral image (HSI) classification. Besides, ensemble learning is a useful way to enhance the classification performance. Therefore, in this study, a new method titled Boosting-CNN is proposed for HSI classification, which fully explored the advantages of deep CNN and ensemble learning. Specifically, several deep CNNs are well-designed to classify HSI. The samples are misclassified in a CNN have more weighs in the following CNN through adaptive boosting. The final classification result is obtained by weighted voting of several CNNs. For HSI classification issue, the number of samples in different classes varies greatly, however, traditional classification methods cannot handle this issue well. In order to address imbalance training samples in HSI classification, soft class balanced loss is proposed to mitigate the influence of imbalance training samples. Experimental results on two popular hyperspectral datasets (i.e., Salinas and Pavia University) show that the proposed method obtain better classification accuracy compared to comparison methods.

In recent years, deep convolutional neural networks (CNNs) have been widely investigated for hyperspectral image (HSI) classification. The CNN-based HSI classifiers obtained good performance under the condition of sufficient training samples. In order to address the problem of limited training samples, in this letter, transfer learning is combined with CNN to address the issue of HSI classification. Pretrained models on large-scale data sets (e.g., ImageNet) can extract the general and discriminative features. Due to the fact that the extracted low-level and mid-level features can be reused for the HSI feature extraction, the CNN-based methods usually obtain good classification performance with insufficient training samples. The ImageNet data set has three channels, while the HSI data set contains hundreds of channels. Therefore, three channels of the HSI are randomly selected to formulate a transferring CNN. Then, several transferring CNNs are combined to establish an ensemble classification system with diversity. Moreover, an improved label smoothing technique is proposed to further improve the classification accuracy of the HSI. Experimental results on two popular hyperspectral data sets [i.e., Indian Pines and Kennedy Space Center (KSC)] show that the transferring CNN ensemble obtains good classification performance compared to the state-of-the-art methods.

In recent years, deep convolutional neural networks (CNNs) have been widely used for hyperspectral image (HSI) classification. The powerful feature extraction capability and high classification performance of CNN highly depend on sufficient training samples. Unfortunately, it is not a common situation because collecting training samples is time-consuming and expensive. In this letter, in order to make the most of deep CNN with limited training samples, dual-path siamese CNN (Dual-SCNN) is proposed for HSI classification. Specifically, the proposed classification framework is a combination of extended morphological profiles, CNN, siamese network, and spectral-spatial feature fusion. In order to solve the problem of insufficiency in hard negative pairs during the training of a siamese network, adversarial training is combined with Dual-SCNN (Dual-SCNN-AT) for HSI classification. Moreover, a data augmentation method titled mixup is combined with Dual-SCNN and Dual-SCNN-AT to further improve the classification performance of HSI. The obtained results on widely used hyperspectral data sets reveal that the proposed methods provide the competitive results in terms of classification accuracy, especially with limited training samples.

Recently, convolutional neural networks (CNNs) have been widely used for light detection and ranging (LiDAR) data classification. Although CNNs achieve good classification performance for LiDAR data classification, a lot of efforts are needed to design a proper architecture. In this letter, the automatic modularized design of CNN is explored for LiDAR data classification for the first time. First, a searchable architecture containing convolution and pooling operations is used to establish the search space. Then, the optimal building block (i.e., cell), which is the basic part of a deep CNN, is obtained from search space by a gradient decent-based algorithm. At last, by stacking several optimal building blocks, a deep CNN can be formulated for LiDAR data classification. Moreover, in order to mitigate the overfitting problem in training a CNN, improved label smoothing and feature regularization are proposed to further improve the classification performance of LiDAR data. The proposed classification models are evaluated on two popular LiDAR data sets (i.e., the Bayview Park and Houston data sets). The experimental results show that the proposed models provide the competitive results compared to the state-of-the-art methods.

Deep convolutional neural networks (CNNs) have shown their outstanding performance in the hyperspectral image (HSI) classification. The success of CNN-based HSI classification relies on the availability sufficient training samples. However, the collection of training samples is expensive and time consuming. Besides, there are many pretrained models on large-scale data sets, which extract the general and discriminative features. The proper reusage of low-level and midlevel representations will significantly improve the HSI classification accuracy. The large-scale ImageNet data set has three channels, but HSI contains hundreds of channels. Therefore, there are several difficulties to simply adapt the pretrained models for the classification of HSIs. In this article, heterogeneous transfer learning for HSI classification is proposed. First, a mapping layer is used to handle the issue of having different numbers of channels. Then, the model architectures and weights of the CNN trained on the ImageNet data sets are used to initialize the model and weights of the HSI classification network. Finally, a well-designed neural network is used to perform the HSI classification task. Furthermore, attention mechanism is used to adjust the feature maps due to the difference between the heterogeneous data sets. Moreover, controlled random sampling is used as another training sample selection method to test the effectiveness of the proposed methods. Experimental results on four popular hyperspectral data sets with two training sample selection strategies show that the transferred CNN obtains better classification accuracy than that of state-of-the-art methods. In addition, the idea of heterogeneous transfer learning may open a new window for further research.

Deep learning-based methods, especially deep convolutional neural networks (CNNs), have shown their effectiveness for hyperspectral image (HSI) classification. In previous deep CNN-based HSI classification methods, a cuboid is empirically determined as the input. The dimensionalities of the cuboid, including height and weight, are crucial to the final classification results. Unfortunately, these superparameters (i.e., the dimensionalities of input cube) are hand-crafted, which means the inputs of a classifier are not optimized according to the specific hyperspectral dataset. In this letter, spatial transformation network (STN) is explored to obtain the optimal input for CNN-based HSI classification for the first time. STN is used to translate, rotate, and scale the original input to obtain optimized input for the following CNN. Moreover, in order to mitigate the overfitting problem in CNN-based HSI classification, DropBlock is introduced as a regularization technique for HSI accurate classification. Compared with dropout, which is a popular regularization technique, DropBlock obtains better classification accuracy. The proposed methods are tested on two widely used hyperspectral data sets (i.e., Salinas and Kennedy Space Center). The obtained experimental results show that the proposed methods provide competitive results compared with state-of-the-art methods including deep CNN-based methods.

Light detection and ranging (LiDAR) is a useful data acquisition technique, which is widely used in a variety of practical applications. The classification of LiDAR-derived rasterized digital surface model (LiDAR-DSM) is a fundamental technique in LiDAR data processing. In recent years, deep learning methods, especially convolutional neural networks (CNNs), have shown their capability in remote sensing areas, including LiDAR data processing. Traditional deep models empirically use a fixed neighborhood system as input to the network. Therefore, the weight and height of the input rectangle may not be optimal. In order to modify such handcrafted setting, a spatial transformation network is used here to identify optimal inputs. The transformed inputs are fed into a well-designed CNN to obtain the final classification results. Furthermore, morphological profiles are combined with spatial transformation CNN to further improve the classification accuracy. The proposed frameworks are tested on two LiDAR-DSMs (i.e., the Recology and Houston data sets). The experimental results show that the proposed models provide competitive results compared to the state-of-the-art methods. Furthermore, the proposed optimal input identification approach can also be found beneficial for other remote sensing applications.

This letter presents a study of vehicle detection in high-resolution images using superpixel segmentation and iterative convolutional neural network strategy. First, a novel superpixel segmentation integrated with multiple local information constraints method is proposed to improve the segmentation results with a low breakage rate. To make training and detection more efficient, we extract meaningful and nonredundant patches based on the centers of the segmented superpixels. For reducing the instability in detection performance because of manual or random selection of samples, a training sample iterative selection strategy based on convolutional neural network is proposed. After a compact training sample subset is obtained from the original entire training set, a representative feature set with high discrimination ability between vehicle and background is extracted from these selected samples for detection. To further avoid overfitting the training and promote the detection efficiency, data augment and a main direction estimation method are used. Comparative experimental results on Toronto data indicated the effectiveness of our proposed method.

Hyperspectral image (HSI) classification is a core task in the remote sensing community, and recently, deep learning-based methods have shown their capability of accurate classification of HSIs. Among the deep learning-based methods, deep convolutional neural networks (CNNs) have been widely used for the HSI classification. In order to obtain a good classification performance, substantial efforts are required to design a proper deep learning architecture. Furthermore, the manually designed architecture may not fit a specific data set very well. In this paper, the idea of automatic CNN for the HSI classification is proposed for the first time. First, a number of operations, including convolution, pooling, identity, and batch normalization, are selected. Then, a gradient descent-based search algorithm is used to effectively find the optimal deep architecture that is evaluated on the validation data set. After that, the best CNN architecture is selected as the model for the HSI classification. Specifically, the automatic 1-D Auto-CNN and 3-D Auto-CNN are used as spectral and spectral-spatial HSI classifiers, respectively. Furthermore, the cutout is introduced as a regularization technique for the HSI spectral-spatial classification to further improve the classification accuracy. The experiments on four widely used hyperspectral data sets (i.e., Salinas, Pavia University, Kennedy Space Center, and Indiana Pines) show that the automatically designed data-dependent CNNs obtain competitive classification accuracy compared with the state-of-the-art methods. In addition, the automatic design of the deep learning architecture opens a new window for future research, showing the huge potential of using neural architectures' optimization capabilities for the accurate HSI classification.

Hyperspectral image (HSI) classification has become a hot topic in the field of remote sensing. In general, the complex characteristics of hyperspectral data make the accurate classification of such data challenging for traditional machine learning methods. In addition, hyperspectral imaging often deals with an inherently nonlinear relation between the captured spectral information and the corresponding materials. In recent years, deep learning has been recognized as a powerful feature-extraction tool to effectively address nonlinear problems and widely used in a number of image processing tasks. Motivated by those successful applications, deep learning has also been introduced to classify HSIs and demonstrated good performance. This survey paper presents a systematic review of deep learning-based HSI classification literatures and compares several strategies for this topic. Specifically, we first summarize the main challenges of HSI classification which cannot be effectively overcome by traditional machine learning methods, and also introduce the advantages of deep learning to handle these problems. Then, we build a framework that divides the corresponding works into spectral-feature networks, spatial-feature networks, and spectral-spatial-feature networks to systematically review the recent achievements in deep learning-based HSI classification. In addition, considering the fact that available training samples in the remote sensing field are usually very limited and training deep networks require a large number of samples, we include some strategies to improve classification performance, which can provide some guidelines for future studies on this topic. Finally, several representative deep learning-based classification methods are conducted on real HSIs in our experiments.

In recent years, deep learning-based methods, especially convolutional neural networks (CNNs), have shown their capabilities in remote sensing data processing. The efficacy of light detection and ranging (LiDAR) has been already proven in a wide variety of research areas. Most of the existing methods do not extract the informative features from LiDAR-derived rasterized digital surface models (LiDAR-DSM) data in a deep manner. In order to utilize the advantages of deep models for the classification of LiDAR-derived features, deep CNN is proposed here to hierarchically extract the robust and discriminant features of the input data. Moreover, morphological profiles and multiattribute profiles (MAPs) are investigated to enrich the inputs of the CNN and further to improve the ultimate classification performance. Furthermore, a new activation function, sigmoid-weighted linear units (SiLUs), is introduced. The proposed frameworks are tested on two LiDAR-DSMs (i.e., Bayview Park and Houston data sets). The MAP-CNNs with SiLU outperform original CNNs by 6.62% and 6.88% in terms of overall accuracy on Bayview Park and Houston data sets, respectively, when the number of training samples of each class is 40.

A generative adversarial network (GAN) usually contains a generative network and a discriminative network in competition with each other. The GAN has shown its capability in a variety of applications. In this paper, the usefulness and effectiveness of GAN for classification of hyperspectral images (HSIs) are explored for the first time. In the proposed GAN, a convolutional neural network (CNN) is designed to discriminate the inputs and another CNN is used to generate so-called fake inputs. The aforementioned CNNs are trained together: the generative CNN tries to generate fake inputs that are as real as possible, and the discriminative CNN tries to classify the real and fake inputs. This kind of adversarial training improves the generalization capability of the discriminative CNN, which is really important when the training samples are limited. Specifically, we propose two schemes: 1) a well-designed 1D-GAN as a spectral classifier and 2) a robust 3D-GAN as a spectral-spatial classifier. Furthermore, the generated adversarial samples are used with real training samples to fine-tune the discriminative CNN, which improves the final classification performance. The proposed classifiers are carried out on three widely used hyperspectral data sets: Salinas, Indiana Pines, and Kennedy Space Center. The obtained results reveal that the proposed models provide competitive results compared to the state-of-the-art methods. In addition, the proposed GANs open new opportunities in the remote sensing community for the challenging task of HSI classification and also reveal the huge potential of GAN-based methods for the analysis of such complex and inherently nonlinear data.

The multisensory fusion of remote sensing data has obtained a great attention in recent years. In this letter, we propose a new feature fusion framework based on deep neural networks (DNNs). The proposed framework employs deep convolutional neural networks (CNNs) to effectively extract features of multi-/hyperspectral and light detection and ranging data. Then, a fully connected DNN is designed to fuse the heterogeneous features obtained by the previous CNNs. Through the aforementioned deep networks, one can extract the discriminant and invariant features of remote sensing data, which are useful for further processing. At last, logistic regression is used to produce the final classification results. Dropout and batch normalization strategies are adopted in the deep fusion framework to further improve classification accuracy. The obtained results reveal that the proposed deep fusion model provides competitive results in terms of classification accuracy. Furthermore, the proposed deep learning idea opens a new window for future remote sensing data fusion.

Recently, the capability of deep learning-based approaches, especially deep convolutional neural networks (CNNs), has been investigated for hyperspectral remote sensing feature extraction (FE) and classification. Due to the large number of learnable parameters in convolutional filters, lots of training samples are needed in deep CNNs to avoid the overfitting problem. On the other hand, Gabor filtering can effectively extract spatial information including edges and textures, which may reduce the FE burden of the CNNs. In this letter, in order to make the most of deep CNN and Gabor filtering, a new strategy, which combines Gabor filters with convolutional filters, is proposed for hyperspectral image classification to mitigate the problem of overfitting. The obtained results reveal that the proposed model provides competitive results in terms of classification accuracy, especially when only a limited number of training samples are available.

It is useful and challenging to analyze and select object features of very high resolution (VHR) remote sensing imagery. The overwhelming majority of existing feature selection methods always concatenate all of the features into a long feature vector and then select features from the vector, ignoring the homogeneity and heterogeneity of underlying feature subspaces. In this paper, we propose a supervised multiview feature selection (SMFS) method. Unlike the existing multiview methods, SMFS requires no prior knowledge of the number of views, and is independent of a prefixed classifier. By utilizing homogeneity and heterogeneity of the data, SMFS employs affinity propagation to automatically decompose features into multiple disjoint and meaningful feature groups or views without any prior knowledge. A group or view consists of homogeneous features, describing a unique data characteristic. Different views represent heterogeneous data characteristics. Then, features are evaluated and selected based on joint ℓ1,2-norm minimization of a loss function and a regularization term. Different from the popular ℓ2,1-norm, joint ℓ1,2-norm enforces the intraview sparsity, instead of interview sparsity. Consequently, a view can be represented by a few representative features in each view, and the information of heterogeneous views can be well kept by the remaining representative features. The experimental results on four VHR satellite images attest to the effectiveness and practicability of SMFS in comparison with single-view algorithms. Furthermore, some discussions are conducted to give insights into homogeneity and heterogeneity of features.

In this letter, a self-improving convolutional neural network (CNN) based method is proposed for the classification of hyperspectral data. This approach solves the so-called curse of dimensionality and the lack of available training samples by iteratively selecting the most informative bands suitable for the designed network via fractional order Darwinian particle swarm optimization. The selected bands are then fed to the classification system to produce the final classification map. Experimental results have been conducted with two well-known hyperspectral data sets: Indian Pines and Pavia University. Results indicate that the proposed approach significantly improves a CNN-based classification method in terms of classification accuracy. In addition, this letter uses the concept of dither for the first time in the remote sensing community to tackle overfitting.

Due to the advantages of deep learning, in this paper, a regularized deep feature extraction (FE) method is presented for hyperspectral image (HSI) classification using a convolutional neural network (CNN). The proposed approach employs several convolutional and pooling layers to extract deep features from HSIs, which are nonlinear, discriminant, and invariant. These features are useful for image classification and target detection. Furthermore, in order to address the common issue of imbalance between high dimensionality and limited availability of training samples for the classification of HSI, a few strategies such as L2 regularization and dropout are investigated to avoid overfitting in class data modeling. More importantly, we propose a 3-D CNN-based FE model with combined regularization to extract effective spectral-spatial features of hyperspectral imagery. Finally, in order to further improve the performance, a virtual sample enhanced method is proposed. The proposed approaches are carried out on three widely used hyperspectral data sets: Indian Pines, University of Pavia, and Kennedy Space Center. The obtained results reveal that the proposed models with sparse constraints provide competitive results to state-of-the-art methods. In addition, the proposed deep FE opens a new window for further research.

Deep convolutional neural networks (CNNs) have shown their great success on image classification. CNNs mainly consist of convolutional and pooling layers, both of which are performed on local image areas without considering the dependence among different image regions. However, such dependence is very important for generating explicit image representation. In contrast, recurrent neural networks (RNNs) are well known for their ability of encoding contextual information in sequential data, and they only require a limited number of network parameters. Thus, we proposed the hierarchical RNNs (HRNNs) to encode the contextual dependence in image representation. In HRNNs, each RNN layer focuses on modeling spatial dependence among image regions from the same scale but different locations. While the cross RNN scale connections target on modeling scale dependencies among regions from the same location but different scales. Specifically, we propose two RNN models: 1) hierarchical simple recurrent network (HSRN), which is fast and has low computational cost and 2) hierarchical long-short term memory recurrent network, which performs better than HSRN with the price of higher computational cost. In this paper, we integrate CNNs with HRNNs, and develop end-to-end convolutional hierarchical RNNs (C-HRNNs) for image classification. C-HRNNs not only utilize the discriminative representation power of CNNs, but also utilize the contextual dependence learning ability of our HRNNs. On four of the most challenging object/scene image classification benchmarks, our C-HRNNs achieve the state-of-the-art results on Places 205, SUN 397, and MIT indoor, and the competitive results on ILSVRC 2012.

Hyperspectral images (HSIs) are often contaminated by noise, in order to remove the image noise efficiently and acquire excellent results. We propose a new denoising method based on 3D sparse coding. Firstly, to make full use of spectral information of hyperspectral data, we extract patches from HSIs and each patch contains the same area of different band. Secondly, we use aforementioned method to extract all patches and train these patches, the dictionary can be obtained, further calculate sparse coefficients. Finally, we can restore the HISs through the dictionary and the sparse coefficients. Experiments are implemented using the HSIs collected by AVIRIS and ROSIS. Results indicate that compared with common 2D sparse coding method, 3D sparse method can effectively improve the restoration performance for both subjective visual and objective evaluation criterion.

In recent years, the specialized destriping methods for remote-sensing image emphasize efficiency and flexibility but lose the particulars in wide stripes. Though many excellent inpainting algorithms have been proposed, most of them are too complex and inefficient for hyperspectral image (HSI). In this paper, we propose a novel automatic completing method which is appropriate for the feature of HSI and easy to be implemented in parallel. The boundary information in broken region is restored primarily based on SVM with the input of the surrounding detected edges. Then we introduce an adaptive morphological filling operator restrained by the boundaries to estimate the pixels in the same position of all bands synchronously. Experimental results show satisfactory performance of the proposed system and its significance on HSI completion.

The paper presents a novel ensemble system which unites Adaboost with multifeature to increase diversity among individual classifiers. Adaboost gives rise to convenience for hyperspectral data classification. To improve the method further, we propose joint Adaboost and multifeature based ensemble (JAME), which assigns different multifeature sets to individual classifiers in Adaboost. Diverse spectral and spatial feature sets are integrated to form multifeature sets. As a result, compared with Adaboost the method has increased the diversity of ensemble system, and better overall accuracies are present. Experiments on hyperspectral data sets reveal that the proposed JAME obtains sound performances comparing with original Adaboost and single classifier.

Hyperspectral images (HSIs) are often contaminated by noise, some spectral bands are highly corrupted that they are usually discarded before processing. To make full use of hyperspectral data, a new bidirectional gradient (BG)-prediction-based HSI junk bands restoration algorithm is proposed. Firstly, according to the field spectral reflectance curves continuity and high spectral resolution instruments, both sides of the junk bands reflectance relative to wavelength gradients can be estimated respectively. Thus, calculate the two estimates of each junk band. Finally, followed by introducing the weighting factor which is inversely proportion to the square of wavelength difference and weighting the two estimates, the results of BG-prediction can be obtained. Experiments are implemented using the HIS collected by airborne visible/infrared imaging spectrometer (AVIRIS). Results indicate that compared with linear prediction, bidirectional gradient prediction can effectively improve the restoration performance, meanwhile the ground classification accuracy of the restored HSIs are improved.

In hyperspectral images, there is abundant spectral information for classification. In most cases, spectral information based methods yield good classification results. However, different objects may have similar spectrums due to similar physical property. Spectral information based classification methods can't give accurate results for the similar physical property among objects belonging to the same main category. On the other hand, as the resolutions of sensors increase in recent years, more spatial information, such as shape and texture information can be extracted and described more precisely. In this paper, we proposed a spatial information aided similar spectral classification. In our method, spatial features, including the pixel shape index (PSI), and the gray-level co-occurrence matrix (GLCM), are extracted for accurate classification. We compared the Bhattachary Distance before and after spatial information being aided and we found that the B Distance was amplified sharply, which indicated that the separability among classes increased. Classification is practiced on two hyperspectral data sets, Kennedy Space Center and Pavia City, and the proposed method is compared with classification based on different features. It is found that each feature makes contribution to classification and the accuracy of the proposed method is the highest among all methods, which certifies the effectiveness of our algorithm.

The existence of nonlinear characteristics in hyperspectral data is considered as an influential factor curtailing the classification accuracy of canonical linear classifier like k-nearest neighbor (k-NN). To deal with the problem, we investigated approaches to combine manifold learning methods and the k-NN classifier to preserve nonlinear characteristics contained in hyperspectral imagery. Then we proposed a Riemannian manifold learning (RML) based k-NN classifier for hyperspectral image classification, which substitutes the Euclidean distances used in canonical kNN by geodesic distances yielded by RML. The experimental results on AVIRIS data show that in most cases, the RML-kNN Classifier accesses higher classification accuracies than canonical k-NN.

3D city models typically consist of thousands of buildings in different types. We usually reconstruct these buildings automatically from high-resolution satellite or airborne imagery. However, for detailed roof reconstruction, 2D information offered by imagery data is not enough while DSM data is necessary. In this paper, we propose a novel level set framework for 3D building models in LOD2 with geometry structure of typical roofs. Local information is introduced towards multiphase and multichannel level set method. Its energy function is minimized when each part of roof data corresponds to the same normal vector as feature values for level set segmentation. The advantage of this method is that for complex building models, roof primitives as well as roof topology graph can be extracted from high-resolution DSM data with high accuracy, evaluated by completeness of segmentation and RMSE of 3D reconstruction. Thus, LOD2 building models can be reconstructed automatically with good performance. The very promising experimental results demonstrate the potentials of our method for large-scale building reconstruction in LOD2.

Advances in sensor and computer technology are revolutionizing the way that remote sensing data with hundreds or even thousands of channels for the same area on the surface of the earth is collected, managed and analyzed. In this paper, the classical Spectral Angle Mapper (SAM) algorithm, which is fit for parallel and distributed computing, is implemented by using Graphic Processing Units (GPU) and distributed cluster respectively to accelerate the computations. A quantitative performance comparison between Compute Unified Device Architecture (CUDA) and Matlab platform is given by analyzing result of different parallel architectures' implementation of the same SAM algorithm.

Vector Quantization (VQ) is widely used for Hyper Spectral Image (HSI) compression and VQ based algorithms yield good results for reducing the amount of the data. However, the VQ based algorithms have the shortcoming of computing expensive. Many fast VQ algorithms have been proposed to reduce the computing complexity, while the algorithms consider the HSI feature rarely. We present a new framework of fast vector quantization for HSI compression, which uses the spectra characteristics of HSI adequately. The breakthrough codebook training method is calculated at the HSI feature domain, which is a low dimension structure without losing significant information, to get much lower complexity. The experimental results demonstrate that the proposed algorithms can reduce the computing time dramatically while keep the comparable reconstruction fidelity.

Achieving high target recognition accuracy is a pursuing and challenging issue for hyperspectral data analysis. The complicated imaging environment and noise interference lead to heterogeneous spectra within the homogeneous object, which makes the current spectral target recognition methods be lack of robustness. In this paper, a robust spectral target recognition method is proposed based on the combined spectral signatures, in which two key techniques are concerned. One is support vector data description (SVDD), which can tolerate the spectral variations of different pixels in the same object. Another is an effective spectral signature combined of spectral reflectance and spectral derivative, which can be robust to data characteristics with different spectral-amplitude variation. The proposed method outperforms the classical methods with only spectral reflective information in term of target recognition accuracy and robustness.

Hyperspectral images (HSI) regularly contain hundreds of bands, which are of different importance in the application. Most HSI compression methods usually deal with most bands in the same way, and they do not take the difference of different bands into consideration, which may cause the loss of important spectral information. In order to preserve the spectral information of interest for applications, a new band-of-interest (BOI)-preserving-based HSI compression method is proposed. The conception of BOI is proposed because some bands are significant in the specific applications, and BOI selection methods are chosen according to application requirements. BOI selection is first performed according to application measurements. Then, BOI information is fed into recursive bidirection prediction (RBP) and set partition in hierarchical trees (SPIHT) compression scheme which uses RBP for spectral decorrelation followed by SPIHT algorithm for coding the resulting decorrelated residual images. More bits are allocated to BOI to preserve BOI by two approaches, respectively. Compress BOI and non-BOI bands directly with low distortion and high distortion, respectively, and compress all bands with low distortion and perform a postcompression truncation. Experiments are implemented with different settings using AVIRIS images. Results indicate that the proposed two methods both can achieve excellent compression efficiency and reconstructed quality. In addition, they can improve the application effect in both material classification and target recognition. Compared with non-BOI compression algorithm, at the compression ratio of 80, the proposed methods improve the classification accuracy by 2% and target recognition accuracy by 9%.

With much richer information than multispectral image, hyperspectral image has been applied in many aspects such as agriculture, environment, military etc. But higher spectral resolution is accompanied by a huge volume of image data, which will result in excessive computing time and data complexity for transmission and storage, so it is necessary to compress hyperspectral image. Because of limit of spatial resolution, some targets that we are interested in are usually in small size and belong to high frequency. In order to preserve such information as much as possible, a hierarchical compression method with protection strategy for hyperspectral image is proposed in this paper, which can protect information of interest (IOI) to some extent, including spatial and spectral IOI. The experimental results show that under the same compression rate, the proposed method can achieve better performance in target detection application than typical SPIHT method.