Recently, unsupervised domain adaptation (UDA) techniques have been introduced for cross-scene hyperspectral image (HSI) classification tasks. These techniques aim to transfer knowledge from labeled source scenes to unlabeled target scenes, addressing the issue of limited supervisory information. However, most UDA methods fail to analyze the variability of domain shifts from different source samples to target ones, thus limiting the domain adaptation effect. To this end, this paper develops a consistency-aware customized learning (CACL) approach for cross-scene HSI classification. Overall, domain-level and class-level distribution alignment are designed separately. The former is implemented by adversarial training between the feature extractor and the domain discriminator. For the latter, the spectral-spatial prototypes of the source and target domains are first dynamically extracted, respectively. Then the prototype-based labels are assigned to the target domain samples, according to the cosine similarity-based cross-domain category prototype matching strategy. Considering that the consistency of the prototype-based labels with the predicted pseudo-labels reflects the degree of domain shifts of the target samples, a customized learning strategy is developed via inter-/intra-domain contrastive learning. With the joint domain-level and fine-grained class-level distribution alignment, the supervised information from the source domain is better migrated to the target domain, improving classification performance. Comprehensive experiments on two single-modal and one multi-modal cross-scene datasets demonstrate the effectiveness of the proposed algorithm.

RGB-infrared (RGB-IR) object detection has attracted significant attention in drone-based applications due to its robustness under all-weather conditions. How to effectively fuse the complementary information in both modalities is one key for accurate object detection. However, the performance is limited by the inherent differences between modalities and the varying illumination conditions across different weather scenarios. Focused on this issue, we propose an illumination-aware multimodal hierarchical fusion network (IMHFNet) for RGB-IR object detection. First, an illumination aware module (IAM) is designed to extract local illumination features from RGB image, which is used to guide the subsequent multimodal feature fusion process. Then, considering the differences in semantic expression and detail representation of different feature layers of multimodal data, we separately design shallow and deep feature fusion strategies. In specific, the shallow feature fusion module is constructed based on convolutional operators and illumination-guided adaptive weight fusion, focusing on capturing and enhancing local detail information. For the deep feature fusion, illumination feature is incorporated as an auxiliary information, to guide the global semantic information integration across different modalities via adopting a transformer structure. In this work, we also construct a new drone-based RGB-IR dataset, named by DroneShip. It contains 4,306 images annotated with 17,054 oriented ship object instances, which covers a wide range of natural illumination conditions from daytime to nighttime. Finally, to validate the effectiveness of the proposed method, we evaluate the IMHFNet on the constructed DroneShip and two publicly available RGB-IR datasets (KAIST and DroneVehicle), which respectively focus on ship, pedestrian and vehicle targets. Experimental results on all three datasets consistently demonstrate the effectiveness and robustness of IMHFNet across diverse scenarios and illumination conditions. The source code of the proposed method will be made publicly available at https://github.com/luting-hnu/DroneShip.

Recently, the Vision Transformer (ViT) becomes a promising model for building extraction from remote sensing images. However, most of these methods ignore the misalignment issue caused by the semantic inconsistency between different layers. To settle this issue, we propose a semantic alignment network (SANet) for building extraction. First, a dense prediction network with the encoder-decoder architecture is developed. The encoder is constituted by stacking multiple ViT blocks to extract multi-level features. Then, the semantic alignment module (SAM) is designed as the decoder to align and fuse low-level and high-level features, by use of semantic flow field estimation. The semantic flow assesses discrepancy between features of adjacent layers to find a flow field that guide the alignment between two-layer feature maps effectively. By this manner, the semantic gap can be narrowed. Experimental results on the two well-known datasets demonstrate the effectiveness of the proposed method.

Remote sensing scene classification has garnered significant interest. However, few studies are dedicated to the classifi-cation of hyperspectral remote sensing scenes. Furthermore, current approaches that are designed for multispectral or visible images are not suitable for hyperspectral remote sensing scene classification. To solve these issues, a spectral-spatial convolutional network is proposed for hyperspectral remote sensing scene classification. First, a multiscale regional growth search method is developed to extract object region of the input data. Then, a three-branch network is designed to extract local spatial, global spatial, and spectral features. Finally, the obtained features are fed into fully connected layer to obtain class scores followed by a decision fusion rule to yield the final classification result. To evaluate the effect of hyperspectral remote sensing scene classification, a hyper-spectral remote sensing scene classification dataset (HRSS-C) is constructed, which consists of 1445 hyperspectral images with 11 different classes. Experiments on the HRSS-C reveal that our method gains superior classification performance with respect to other advanced approaches.

How to learn the most valuable and useful features in convolutional neural networks (CNNs) is the key for accurate hyperspectral image classification (HSIC). Focused on this issue, we developed a local–global gated CNN (LGG-CNN), in this letter. The core is the simultaneous construction of local and global gated convolution blocks, with the aim to select highly discriminative information and filtering redundant information in hyperspectral images (HSIs). Different from traditional CNN methods treating all spectral–spatial features equally, the gated convolutions help in learning a normalized soft mask to guide the network to focus on valid features and neglect the invalid ones. Here, based on the CNN backbone, multilayer local features are first learned via gated convolutional architecture, which mainly consists of convolution operators and nonlinearly activation functions. At the same time, a global gated block (GGB) is designed to conduct feature serialization-mapping-patching operations, to learn global features from deeper layers with larger receptive fields. As a result, the local/GGBs can dynamically learn discriminative feature selection mechanisms for each channel at each spatial location. Then, the local and global features are fused at both the feature-level and decision-level. In this manner, the effective fusion of features by the multilayer LGG convolution blocks enables spatial interaction across layers, leading to further improvement in classification accuracy. Extensive experiments on three benchmark HSIC datasets demonstrate the superiority of LGG-CNN over some state-of-the-art methods. The source code of the proposed method is available athttps://github.com/Ding-Kexin/LGG-CNN.

Semi-supervised classification of remote sensing hyperspectral image (HSI) aims at exploiting both labeled and unlabeled samples for accurate land cover recognition. However, imbalanced data distribution and different classification difficulties negatively affect classification performance. Focused on this, a novel dual-stream class-adaptive network (DSCA-Net) is proposed for semi-supervised HSI classification, in this paper. First, a superpixel-guided label propagation module is introduced to alleviate the negative effect of imbalanced data distribution. Specifically, approximate estimation of labels for unlabeled samples is achieved via superpixel-wise similarity measure and label propagation, so that equal sampling is applied to each class. Then, a consistency regularization-based dual-stream network is constructed, which shares the same encoder for feature representation of either labeled or unlabeled samples. Based on this, two distinct classifiers are designed to force similar predictions can be achieved for various perturbed versions of the same unlabeled sample, thereby allowing unlabeled samples to train the model in a supervised manner. Finally, since different classes always have various degrees of learning difficulty, equal treatment may lead to overfitting of “easy” classes and biased prediction of “hard” classes. Unlike the traditional selection of unlabeled samples with a fixed threshold, dynamic class-adaptive thresholds are calculated according to the learning status of the model. In this manner, a higher threshold is assigned to “easy” classes to reduce sample redundancy, and a lower threshold is set for “hard” classes to select more samples. Experiment results demonstrate the effectiveness and superiority of the proposed method. Codes are available at https://github.com/luting-hnu/DSCA-Net.

Building extraction is a challenging research direction in remote sensing image (RSI) interpretation. Due to the fact that a building has not only its own local structures but also similar architectural styles with other buildings located in a global area (e.g., street or community), fusing local and global features becomes a promising way to improve performance of building extraction. Focused on this, we propose a new complementarity-aware local-global feature fusion network (CLGFF-Net) by integrating a convolutional branch and a Transformer branch. The two branches respectively capture local patterns and global long-range dependencies of RSIs, thereby leading to highly complementary features. To dig out the implicit complementary information for fusion, we develop a complementarity-aware fusion module (CFM) which separates shared features (SFs) and distinct features (DFs) between two branches, by building a commonalities analysis path and two difference analysis paths. Meanwhile, to make sure the similarity of SFs and dissimilarity of DFs, a triplet loss function is designed to enforce the distances between SFs to be near and DFs to be far. By this way, complementary information can be explicitly included in DFs and is adaptively exchanged between two branches for fusion. Besides, since multilayer features in each branch generally convey different-level semantic information, a multi-layer fusion scheme (MLFS) is designed to fuse them by introducing cross-layer connections and gate mechanism. By coupling CFMs with MLFS, the abilities in characterizing local and global context information, as well as different-level semantic information, can be fully exploited for better mapping of complicated building objects. Experimental results demonstrate the effectiveness of our proposed method.

Deep learning has been a powerful tool for hyperspectral image (HSI) classification. However, it is still an open issue to effectively learn highly discriminative features from the HSI, due to the high-dimensionality and complex spectral-spatial characteristics. To settle this issue, we propose a new band-grouping guided multi-attention module for the performance promotion of spectral-spatial feature learning. First, based on the fact of high relevance between adjacent spectral bands and low dependencies across long-range ones, all the spectral bands are adaptively divided into multiple non-overlapping groups where relevant bands are included. The advantage is to reduce the spectral dimension and data complexity when processing and analyzing each group. Then, a multi-attention mechanism, which not only explore the intra-group salient information but also propagate the inter-group difference information, is embedded into the convolutional neural networks to learn group-specific spectral-spatial features. By emphasizing useful spectral/spatial information and squeezing useless information with attention mechanism, the severability of learned features is enhanced. Based on this module, a spectral-spatial classification network is built, named by grouped multi-attention network (GMA-Net). The GMA-Net contains a two-branch architecture, i.e., pixel-wise spectral feature learning and patch-wise spectral-spatial feature learning. Via fusing the features from two branches, the complementary and discriminative features provided by pixel-wise and patch-wise learning manner can be integrated to further boost classification performance. Experimental results demonstrate that the proposed method is superior than several state-of-the-art approaches. Codes are available at: https://github.com/luting-hnu.

Deep learning based methods are very popular for hyperspectral image classification. However, those methods usually ignore the fact that discriminative information lies on specific spatial positions and spectral bands. To solve this problem, we introduce the attention mechanism, and propose a cross-layer multi-attention guided classification network (CLMA-Net) for HSIs. First, a backbone network, which is a two-branch convolutional neural network, is developed to extract spectral and spatial features. Then, cross-layer multi-attention modules, which integrate attention information of multiple convolutional layers, are embedded into two branches. As a result, spectral and spatial features are optimized by making the network attend to interested parts. Finally, spectral and spatial features are concatenated and used to predict class label by a fully connected layer. Experimental results demonstrate the effectiveness of the proposed method. The code will be available at https://github.com/mengkai-liu/CLMA-Net.

Building change detection is a very important application in the field of remote sensing. Recently, deep learning (DL) has been introduced to solve the change detection task and achieved good performance, mainly due to the capability of automatically learning deep features. However, the lack of using prior knowledge (e.g., edge structure information) leads to inaccurate detection results, especially in the areas of building boundaries. To solve this problem, an end-to-end DL method for building change detection, named by edge-guided recurrent convolutional neural network (EGRCNN), is proposed in this article. The main idea is to incorporate both discriminative information and edge structure prior in one framework to improve change detection results, especially to generate more accurate building boundaries. First, a siamese convolutional neural network is trained to simultaneously extract primary multilevel features from multitemporal images. Then, a difference analysis module (DAM) is introduced to further produce discriminative features, which is constructed based on the basic long short-term memory module. Finally, both the discriminative features and the estimated edge structure information are jointly exploited to predict building change map. On one hand, the proposed DAM helps to enhance the discrimination between the changed and unchanged regions. On the other hand, the prior edge information is used to push the predicted changed buildings to preserve the original structure, which can further improve the accuracy of building change detection. Experimental results demonstrate that the performance of the proposed method outperforms several state-of-the-art approaches, in terms of objective metrics and visual comparison results.

Hyperspectral images (HSI) contain rich spatial and spectral detail information, while light detection and ranging (LiDAR) data can provide the elevation information. Thus, the fusion of HSI and LiDAR data can help for more accurate image classification, which becomes a hot research topic. However, it is difficult to capture complex local and global spatial-spectral associations, meanwhile, how to build an effective interaction between multi-modal data is another important issue. To this end, a novel global-local transformer network (GLT-Net) is proposed for the joint classification of HSI and LiDAR data, in this paper. The main idea is to fully exploit the advantage of the convolution operator in characterizing locally correlated features and the promising capability of transformer architecture in learning long-range dependencies. Moreover, multi-scale feature fusion and probabilistic decision fusion strategies are also designed in one framework, in order to further improve classification performance. Here, the proposed GLT-Net mainly consists of multi-scale local spatial feature learning, global spectral feature learning, and global-local feature fusion classification. In specific, multi-modal image cubes of different sizes are firstly extracted and sent into convolutional neural networks (CNNs) to learn local spatial features, which is followed by multi-modal information propagation and spatial-attention guided multi-scale feature fusion. Afterwards, by considering spectral feature channels from a sequential perspective, vision transformers are introduced to model the global spectral dependencies. Finally, multiple class estimations based on local and global features are integrated via a probabilistic decision fusion strategy. In this way, complementary information of multi-modal data as well as local/global spectral-spatial information can be fully mined and jointly utilized. Extensive experiments on three popular HSI and LiDAR datasets demonstrate that the proposed method performs superiority over state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/GLT-Net.

In recent years, deep learning presents a promising performance in hyperspectral image (HSI) classification, due to the powerful capability of automatically learning deep semantic characteristics of images. However, it is still difficult to learn highly discriminative features when limited samples are available for training a deep network. Focused on this issue, a novel end-to-end supervised contrastive learning network (SCL-Net) for spectral-spatial classification is proposed, in this paper. Instead of learning features of the individual sample, the supervised contrastive learning is introduced to capture the similarity and dissimilarity distribution properties of sample pairs in feature representation space. In this way, the need for plenty of training samples will be alleviated while an effective network training mechanism is provided for learning highly separative features. Here, the SCL-Net mainly consists of one pair-wise contrastive learning (PCL) sub-network and one multi-level spectral-spatial information fusion (MLSIF) sub-network. For the PCL sub-network, spectral vectors are projected into deep spectral features based on convolutional operators, which are then followed by distance evaluation between “positive” pairs of similar samples and “negative” pairs of dissimilar ones. Then, a spectral distance matrix is constructed to push the network to gradually learn better features of higher intra-class compactness and inter-class dispersion. For the MLSIF sub-network, a hybrid feature-decision fusion strategy is designed, where spatial and spectral features are jointly exploited to further boost classification performance. In specific, the feature fusion is conducted by connecting low/mid/high-level spectral and spatial features via weighting, while multiple class estimations based on multi-level fusion features are adaptively integrated via probabilistic decision fusion. Overall, these two sub-networks are collaboratively trained in one framework, by optimizing a defined joint loss function consisting of a contrastive loss and a cross-entropy loss. Compared with several state-of-the-art methods, the proposed method yields a superior classification performance in terms of both objective metrics and visual performance.

Exploring effective spectral–spatial feature extraction methods is one of the most focused problems in current hyperspectral image (HSI) classification research. However, complex spectral–spatial structure characteristics in HSIs, e.g., shape-variable spatial structure and nonlinear spectral structure, are difficult to be effectively extracted and jointly represented. To overcome this issue, a novel superpixel-based Brownian descriptor (SBD) method for HSI classification is proposed in this article. In specific, superpixel segmentation is first used to extract shape-adaptive spatial structure information from dimension-reduced HSI, leading to generate nonoverlapping homogeneous 3-D image blocks. Then, similar pixels within the 3-D image block are jointly represented by a new local spectral–spatial feature based on the Brownian descriptor (BD). This is the first time that the BD is introduced to measure both linear and nonlinear correlations among different spectral bands in HSI. On one hand, the integration of superpixel and BD helps to provide much richer and more valuable information for better discrimination between different categories. On the other hand, the SBD can effectively represent the internal structure characters within each 3-D image block of different spatial shapes by a symmetric positive definite matrix of a united form. Finally, considering that the SBD lies on the Riemannian manifold space, a log-Euclidean kernel sparse representation (LKSR) classifier is introduced to obtain the classification results. Experimental results on three widely used real hyperspectral datasets indicate the performance superiority of the proposed SBD method over several state-of-the-art techniques.

For hyperspectral image (HSI) classification, it is an challenging problem to learn highly discriminative features, since the complex local/non-local spatial-spectral association is difficult to be accurately characterized. Focused on this issue, a novel superpixel-level hybrid discriminant analysis (SHDA) method is proposed, in this paper. Here, the SHDA method takes advantage of superpixel’s merit in characterizing spatial-spectral shape-adaptive structure and the powerful capability of discriminant analysis in enhancing class-separability to learn the feature representation. Moreover, the local/non-local spatial-spectral correlation information among/between superpixels is effectively excavated and fused in one framework to further improve the classification performance of features. This is achieved by first designing two specific discriminant analysis modules, i.e., superpixel-level local discriminant analysis (SLDA) and superpixel-level non-local discriminant analysis (SNDA). In the SLDA, adaptively weighted scatter matrices are defined to characterize the local spectral similarity within each superpixel and the discrepancy among adjacent superpixels. In the SNDA, superpixel-level graphs are built for capturing the non-local contextual information, where the weights of graphs are estimated based on the most similar and dissimilar superpixels. Then, the SLDA and the SNDA are effectively fused to construct the total intra/inter-superpixel scatter matrices. Finally, a joint projection transformation is obtained via solving a simple generalized eigenvalue problem. By this way, the HSI data can be projected from a high-dimensional data space into a low-dimensional feature space, where different classes of land-covers can be more accurately distinguished. Experimental results on three real hyperspectral data sets indicate that the proposed SHDA method outperforms several state-of-the-art techniques.

Traditional hyperspectral imaging technique obtains numerous hyperspectral images (HSIs) with hundreds of spectral bands, leading to high cost in data acquisition, transmission, and storage. Compressed sensing (CS) theory provides a new imaging mechanism, which relies on the assumption that signals can be sparsely represented over a dictionary. By the CS imaging technique, original HSIs can be approximately reconstructed from only a few sampled measurements. In this article, a novel context-aware CS (CACS) method for HSIs is proposed by incorporating contextual prior to the dictionary learning and the sparse reconstruction. First, a patch-based online dictionary learning (ODL) algorithm is developed by introducing a joint sparse constraint. On the one hand, the online dictionary learning mechanism enables a more adaptive representation of HSIs with different scenes than using fixed-basis-based dictionaries, e.g., the discrete cosine transform (DCT) and the discrete wavelet transform dictionaries. On the other hand, the introduced joint sparse constraint promotes the learned dictionary to more sparsely and structurally represent spectral pixels. Then, with the well-learned dictionary, a weighted smoothing regularization is introduced to develop a new sparse reconstruction model. Considering the high spectral-spatial similarity of pixels in a neighborhood, the new sparse reconstruction model will encourage a locally smoothing reconstruction result. In this way, the spectral-spatial structures of the HSI can be well preserved, while possible artifacts can be effectively reduced. Experimental results demonstrate the superiority of the proposed method over some state-of-the-art hyperspectral compressive imaging methods.

Compressed sensing (CS) technique contributes to reduce the burden of storage and transmission for hyperspectral images (HSIs) which are large 3D data cubes. Previous CS methods usually adopt sensing matrixes whose column number is equal to the length of the signal to sample data. As the length of the signal increases, the sensing matrix can be very large, especially for high-dimensional hyperspectral data. To overcome the drawback, a new semi-tensor based CS (ST-CS) method is proposed to for HSIs. In the sampling model, we construct a semi-tensor sensing matrix whose column number is much smaller than the length of spectral pixel. Then, the semi-tensor product, which breaks the dimension matching condition for matrix multiplication, is applied to the data sampling. The lower-dimensional sensing matrixes can reduce the data storage in the onboard system. In the sparse coding and reconstruction model, the spatial correlation of spectral pixels is exploited by introducing a regularization. The regularization tries to push reconstructed neighboring pixels to be similar. As a consequence, some unsuccessfully reconstructed pixels may be corrected by the use of neighbor information. Furthermore, the spatial structure of the HSI can be better reconstructed. Experimental results show the effectiveness of the proposed method.

Sparse representation (SR) has been successfully used in the classification of hyperspectral images (HSIs) by representing HSI pixels over a dictionary and yielding discriminative sparse coefficients. Most of SR-based classification methods construct the dictionary by directly using some labeled pixels as atoms. Such dictionary can lead to inefficient SR for large-sized HSIs, and may be incomplete when the number of labeled pixels is less than the number of spectral bands. This paper proposes a contextual online dictionary learning (DL) method for HSIs classification, which learns a dictionary over the whole image rather than few labeled pixels. The proposed method can effectively and efficiently improve the adaptive representation capability of different pixels with an online learning mechanism. Specifically, the contextual characteristics of the HSI are integrated with discriminative spectral information for online DL, i.e., pushing similar pixels in neighborhood to share similar sparse coefficients with respect to the well-learned dictionary. By this way, the obtained sparse coefficients are structured and discriminative. Finally, a traditional classifier, i.e., the linear support vector machine, is applied to the sparse coefficients, and the final classification results are obtained. Experimental results on real HSIs show the effectiveness of the proposed method.

Sparse representation (SR) based hyperspectral image (HSI) classification is a rapidly evolving research topic. How to construct an optimized dictionary to better characterize spectral-spatial features of HSI is an important problem. In this paper, a novel spectral-spatial online dictionary learning (SSODL) method for HSI classification is proposed. The main idea is to learn a complete and discriminative dictionary by exploiting both spatial and spectral information all over the whole image. Rather than only using training samples for dictionary construction, the online dictionary learning (ODL) mechanism can effectively improve the adaptive representation capability of different pixels. Specifically, the contextual characteristics of HSI are integrated with discriminative spectral information for the ODL, i.e., pushing similar pixels in neighborhood to share similar sparse coefficients w.r.t. the well learnt dictionary. By this way, the yielding sparse coefficients are structured and discriminative. Finally, a traditional classifier, i.e., linear support vector mechine (SVM), is applied to the sparse coefficients and the final classification results are obtained. Experimental results on real HSIs show the effectiveness of the proposed method.

Sparse representation (SR) can transform spectral signatures of hyperspectral pixels into sparse coefficients with very few nonzero entries, which can efficiently be used for compression. In this paper, a spectral-spatial adaptive SR (SSASR) method is proposed for hyperspectral image (HSI) compression by taking advantage of the spectral and spatial information of HSIs. First, we construct superpixels, i.e., homogeneous regions with adaptive sizes and shapes, to describe HSIs. Since homogeneous regions usually consist of similar pixels, pixels within each superpixel will be similar and share similar spectral signatures. Then, the spectral signatures of each superpixel can be simultaneously coded in the SR model to exploit their joint sparsity. Since different superpixels generally have different performances of SR, their rate-distortion performances in the sparse coding will be different. To achieve the best possible overall rate-distortion performance, an adaptive coding scheme is introduced to adaptively assign distortions to superpixels. Finally, the obtained sparse coefficients are quantized and entropy coded and constitute the final bitstream with the coded superpixel map. The experimental results over several HSIs show that the proposed SSASR method outperforms some state-of-the-art HSI compression methods in terms of the rate-distortion and spectral fidelity performances.

Recently, the superpixel segmentation is introduced into the hyperspectral image (HSI) classification to exploit the spatial information. However, the size of superpixels influences the classification significantly because small superpixels can not provide enough spatial information and large superpixels generally result in error segmentation. The error segmentation is irreversible and intolerable, so the size of superpixels tends to be small. This paper proposes a hyperspectral unmixing based superpixel merging criterion to merge small su-perpixels and thus make use of the spatial information. The spatial information is then incorporated into the joint sparsity model for the spectral-spatial classification. Experimental results demonstrate the superiority of the proposed method over some widely used classification methods.

This paper proposes a new spectral-spatial hyperspectral classification method named the shape-adaptive sparse representation (SASR). The fixed window is not suitable for all pixels of hyperspectral image (HSI) to search local similar regions. In order to overcome the drawback, we propose to apply the shape-adaptive algorithm to exploit the contextual spatial information of HSI. Furthermore, the hyperspectral classification is implemented by incorporating the spatial contextual information of HSI into the sparse representation classification model. Experimental results demonstrate the superiority of the proposed SASR method over both classical and state-of-the-art approaches.