Natural disasters pose a threat to the safety of human life and buildings. Rapid and accurate building damage assessment (BDA) on remote sensing images is crucial for disaster response and recovery. However, most methods are constructed on the ideal co-registered bitemporal remote sensing images, neglecting the misalignment in practice. In this paper, we propose a novel building damage assessment method, termed FlowMamba, which can effectively handle the offset between the pre- and post-disaster images in BDA task. Specifically, a vision mamba backbone with four stages is utilized to extract multi scale features from the pre- and post-disaster images. Then, a differential optical flow alignment module is designed to estimate shift matrix to align pre- and post-disaster features. Furthermore, a category distance-aware loss function is tailored for the BDA task, which replaces fixed binary values of the penalty factors in to soft values of inter-class distance. Extensive experiments on the xBD dataset, the BRIGHT dataset and four out-of-distribution disaster scenarios validate the robustness and effectiveness of the proposed FlowMamba. Our code is available at https://github.com/flying318/FlowMamba.

Visible and infrared light images reflect object characteristics in different aspects, which has attracted much attention for object detection in recent years. Nevertheless, the existing multimodal detection networks may fail in the absence of modality. In order to address this problem, a new cross-modal knowledge distillation network (CMKD-net) is proposed for oriented object detection in visible and infrared images. In brief, a teacher-student (T-S) learning network is constructed, where the T-network aims to learn a discriminative feature representation from multimodal images and then guides the S-network training with incomplete modality. Here, multi-dimensional feature distillation (MDFD) and inter-instance relation distillation (IIRD) are designed for cross-modal knowledge propagation. Specifically, the MDFD considers pushing the T-S networks to learn a similar data distribution and feature representation through channel-spatial dimensional feature consistency constraints. The IIRD contributes to retaining the relation structure between individual targets in multimodal images via inter-instance relation modeling and similarity distance measurement. Moreover, to avoid the bias of feature extraction caused by discrete quantization in traditional pooling operations, a rotation-adaptive RoI Pooling (RA-RoI Pooling) is introduced by calculating the continuous double integral within each bin of oriented objects. Ablation experiments and comparison experiments on the VEDAI and DroneVehicle datasets can demonstrate the effectiveness of the proposed CMKD-net.

Cross-scene hyperspectral image classification aims to identify a new scene in target domain via learned knowledge from source domain using limited training samples. Existing cross-scene alignment approaches focus on aligning the global feature distribution between the source and target domains while overlooking the fine-grained alignment at different levels. Moreover, they mainly use Transformer architectures to model long-range dependencies across different channels but confront efficiency challenges due to their quadratic complexity, which limits classification performance in unsupervised domain adaptation tasks. To address these issues, a new domain-adaptive Mamba (DAMamba) is proposed for cross-scene hyperspectral image classification. First, a spectral-spatial Mamba is developed to extract high-order semantic features from the input data. Then, a domain-invariant prototype alignment method is proposed from three perspectives, i.e., intra-domain, inter-domain, and mini-batch, to produce reliable pseudo-labels and mitigate the spectral shift between the source and target domains. Finally, a fully connected layer is applied to the aligned features in the target domain to obtain the final classification results. Extensive evaluations across diverse cross-scene datasets demonstrate that our DAMamba outperforms existing state-of-the-art methods in classification accuracy and computing time. The code of this paper is available at https://github.com/PuhongDuan/DAMamba.

Multi-source remote sensing data classification refers to the process of categorizing ground objects by integrating complementary strengths of multiple remote sensing data, such as hyperspectral image (HSI), light detection and ranging (LiDAR) and synthetic aperture radar (SAR) data. However, current Mamba-based multisource remote sensing data classification approaches rely on fixed scanning patterns that are inadequate in characterizing spectral-spatial information. Additionally, current fusion techniques adopt concatenation or attention-based fusion rules without considering the complementary characteristics between different modalities. To address these limitations, we propose a spectral-spatial dynamic scan Mamba (SDSM) for multi-source remote sensing data classification. Specifically, a dynamic scan Mamba network is proposed to extract the spectral-spatial features of multi-source remote sensing data, in which a dynamic scan module is designed to adaptively capture the important spatial and spectral information. Furthermore, a bidirectional cross-modal fusion rule is proposed to merge the extracted features, in which a global-local frequency feature extraction module is designed to extract the salient structural features of multi-source remote sensing data as clues to guide heterogeneous feature fusion. Comprehensive experiments on four multi-source remote sensing datasets, i.e., MUUFL, Augsburg, Italy and Yellow River, demonstrate that the proposed method outperforms other state-of-the-art methods with respect to quantitative and qualitative results. The code of this article is available at https://github.com/PuhongDuan/SDSM.

Cross-view localization aims to estimate the precise position and orientation of a ground-view image by aligning it with satellite imagery. However, existing homography-based methods are typically evaluated under limited orientation noise (±45°) and exhibit limited refinement capability, as their local correlation-based refinement relies on a reasonably good initial orientation estimate. To address these limitations, we propose a fine-grained cross-view localization method based on orientation-guided homography (OGH-Net), whose core idea is to predict an initial orientation prior that explicitly guides subsequent homography refinement. Specifically, we first design a hybrid bird's-eye-view (BEV) transformation to generate BEV images with preserved central geometry and expanded coverage. Then, we introduce a lightweight orientation-prior module that provides a coarse yaw estimate across the full ±180° range. Finally, we develop a multiscale iterative homography module that progressively refines the projection matrix through hierarchical iterations across multiple feature resolutions. Under cross-area, unknown-orientation conditions, it reduces mean localization error by 11% and mean orientation error by 27% on VIGOR, and further reduces mean localization error by 27% on KITTI compared with previous state-of-the-art methods. Moreover, OGH-Net runs in real time at up to 107 FPS on a single RTX 3090 GPU, offering a favorable trade-off between accuracy and efficiency. The code and trained models will be released at https://github.com/YC-Zhang2025/OGH-Net.

Segmenting ultra-high-resolution (UHR) images poses a significant challenge due to constraints on GPU memory, leading to a trade-off between detailed local information and a comprehensive contextual understanding. Current UHR methods often employ a multi-branch encoder to handle local and contextual information, which can be memory-intensive. To address the need for both high accuracy and low memory usage in processing UHR images, we introduce a memory-efficient semantic segmentation approach by squeezing context information into local patches (SCPSeg). Our method integrates the processing of local and contextual information within a single-branch encoder. Specifically, we introduce a context squeezing module (CSM) designed to compress global context details into local patches, enabling segmentation networks to perceive broader image contexts. Additionally, we propose a super-resolution guided local feature alignment (LFA) technique to improve segmentation precision by aligning local feature relationships. This approach calculates similarities within sliding windows, avoiding heavy computational costs during the training phase. We evaluate the effectiveness of our proposed method on four widely used UHR segmentation benchmarks. Experimental results demonstrate that our approach enhances UHR segmentation accuracy without incurring additional memory overhead during the inference stage. The code is available at https://github.com/StuLiu/SCPSeg.

Processing high-dimensional data cubes and developing high-performance classifiers are core objectives in the field of hyperspectral image classification (HSIC). Superpixel-based methods are widely used in HSIC due to their efficacy in reducing redundant information and enhancing local features. However, imprecise segmentation, especially in complex structures and textures of hyperspectral images (HSIs), may lead to inconsistencies in the regions extracted by superpixels and the boundaries between different ground objects. Such inconsistencies significantly degrade the classification performance of HSIs. Alternatively, when parameter settings are inaccurate, edge-aware feature extraction methods often introduce sharpening artifacts at the image boundaries, resulting in a decrease in classification accuracy. To effectively address these challenges, we propose a novel probabilistic fusion method for HSIC. This method consists of the following stages. First, spatial information is extracted by a multiscale superpixel segmentation method and then probabilistically optimized by the extended random walk (ERW) method. Next, semantic-aware structural features (S2Fs) are extracted along with edge information of different objects. Lastly, a probabilistic framework is proposed to fuse the class probabilities of superpixel-based spatial information and semantic-aware structural features. Experimental results on three real datasets show state-of-the-art classification performance, even with limited training sets.

Hyperspectral images (HSIs) are often affected by noise originating from both internal imaging mechanisms and external environmental factors. Therefore, denoising serves as a crucial preprocessing step for HSIs. In real-world scenarios, HSI denoising is particularly challenging due to the complex and band-dependent nature of noise. Current Mamba-based models, although capable of sequential modeling, are highly sensitive to input spectral order and still suffer from information loss over relatively long sequences, potentially leading to local over-sharpening and spectral distortion. To address these issues, we propose a novel linear attention Mamba (LaMamba) for HSI denoising. To capture the intrinsic nature of HSI, a 3D selective scan mechanism is designed to convert the input HSI into spectral-spatial continuous sequences using six bidirectional scan orders. Additionally, a linear attention state space model is proposed to capture long-range correlation. Experimental results on both synthetic and real-world hyperspectral datasets demonstrate that our model significantly outperforms other advanced methods in reconstructing spectral fidelity and spatial visual effect. The code is released at https://github.com/PuhongDuan/LaMamba.

Remote sensing scene classification (RSSC) plays a vital role in a variety of applications and has attracted much more attention. In recent years, much progress has been made to release diverse datasets or develop all kinds of techniques for scene classification of multispectral remote sensing images. Nevertheless, very few studies have focused on hyperspectral image scene classification. Moreover, the existing scene classification approaches fail to fully employ the rich spectral information of the input images, which cannot achieve satisfactory performance for hyperspectral images. To alleviate these issues, this work proposes a spectral-spatial fusion network (SSFNet) for hyperspectral RSSC (HRSSC). First, a multiscale regional growth search (MSRGS) method is designed to extract salient object regions from the hyperspectral remote sensing scene. Then, a three-stream network architecture is proposed to extract the global spatial, local spatial, and spectral features, respectively. Finally, the fully connected layer is performed on the extracted features to obtain a class score followed by a decision fusion scheme to generate the final classification result. To evaluate the effectiveness of the proposed SSFNet, we created a publicly available benchmark for the HRSSC dataset, which contains 1445 hyperspectral images, covering 11 scene classes. Experiments on the HRSSC database claim that the proposed SSFNet can attain superior classification performance with respect to other state-of-the-art scene classification techniques. The code of the proposed SSFNet will be available athttps://github.com/PuhongDuan/SSFNet.

Transformer models have achieved remarkable success in hyperspectral image classification (HSIC) owing to their strong global modeling capability. However, their quadratic complexity significantly limits their computational efficiency. Recently, Mamba has been applied to HSIC because of its linear complexity, yet it still suffers from an imbalance between global and local modeling. To overcome these challenges, this paper proposes a novel Learnable Kernel and Mamba with Spatial-Spectral Attention Fusion (LKMA) framework, which enables the extraction of global-local spatial-spectral features (SSF) while enhancing edge feature representation. For local feature extraction, the proposed Multi-Scale Spatial-Spectral Feature Generation (MSSFG) module captures local SSF by employing multi-scale learnable dilation convolutions for spatial features and multi-scale dilation convolutions for spectral features. For global feature extraction, a Global Hidden Mixing Mamba (GHMM) module is introduced, which projects hyperspectral image (HSI) features from the feature space to the hidden state space via a hidden state mixing mechanism. This enables the model to capture contextual semantic information and local details from the HSI. To further explore the synergistic effect between spatial and spectral information, the Spatial-Spectral Attention Fusion (SSAF) module integrates semantic information across multiple feature groups by combining Semantic Grouped Spatial Attention (SGSA) and Progressive Spectral Self-Attention (PSSA), enhancing spatial-spectral representations. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art approaches for HSIC.

Hyperspectral image (HSI) encompasses abundant spatial and spectral details, while Light Detection and Ranging (LiDAR) delivers precise elevation data. The amalgamation of HSI and LiDAR data significantly improves the precision of image classification. However, most methods focus solely on spatial features while neglecting frequency domain information, limiting the ability of deep models to characterize land cover. Furthermore, how to establish a sufficient interaction between different modalities is also an important issue. In this paper, we propose a novel multiscale cross-domain fusion network (MCFNet) for joint classification of HSI and LiDAR data. The main idea is that the wavelet transform can provide details at different resolutions simultaneously, supplementing spatial domain information and enriching feature representation. In addition, the multimodal fusion module (MFM) guided by HSI and the cross-domain fusion module (CDFM) strategy are developed to integrate features from diverse modalities and domains, respectively. Specifically, frequency domain features are extracted by discrete wavelet transform, and spatial domain features of the image are captured through a set of convolution operations. Then interactive fusion is performed by MFM and CDFM, and finally the integrated features are categorized using a classification module. Extensive experiments on three widely-used HSI and LiDAR datasets indicate that MCFNet outperforms the SOTA methods. The code will be available at https://github.com/MSFLabX/MCFNet.

Cross-domain image segmentation plays a crucial role in the field of remote sensing. Current approaches often rely on a mean-teacher model that is integrated from student models to guide the training of the student model itself. However, the feature space of the mean-teacher model exhibits significant domain discrepancy and considerable class overlap, which results in suboptimal performance. Motivated by the idea of learning from stronger teachers, we introduce a robust domain adaptation method called LFMDA. This novel approach is the first to explicitly enhance cross-domain semantic segmentation performance by leveraging vision foundation models (VFMs) within remote sensing applications. Specifically, we propose a prototypical contrastive knowledge distillation loss (PCD) that enables the student model to produce domain-invariant yet category-discriminative features by distilling knowledge from a domain-generalized VFM teacher. Additionally, we introduce a local region homogenization strategy (LRH) to generate high-quality and high-quantity pseudo-labels by incorporating a Segment Anything Model (SAM). Extensive empirical evaluations demonstrate that our method outperforms existing approaches, setting a new state-of-the-art (SOTA) method in domain-adaptive remote sensing image segmentation. The code is available at https://github.com/StuLiu/LFMDA.

Camouflaged object detection (COD) aims to discover objects that are seamlessly embedded in the environment. Existing COD methods have made significant progress by typically representing features in a discrete way with arrays of pixels. However, limited by discrete representation, these methods need to align features of different scales during decoding, which causes some subtle discriminative clues to become blurred. This is a huge blow to the task of identifying camouflaged objects from clear subtle clues. To address this issue, we propose a novel continuous feature representation network (CFRN), which aims to represent features of different scales as a continuous function for COD. Specifically, a Swin transformer encoder is first exploited to explore the global context between camouflaged objects and the background. Then, an object-focusing module (OFM) deployed layer by layer is designed to deeply mine subtle discriminative clues, thereby highlighting the body of camouflaged objects and suppressing other distracting objects at different scales. Finally, a novel frequency-based implicit feature decoder (FIFD) is proposed, which directly decodes the predictions at arbitrary coordinates in the continuous function with implicit neural representations, thus propagating clearer discriminative clues. Extensive experiments on four challenging COD benchmarks demonstrate that our method significantly outperforms state-of-the-art methods. The source code will be available at https://github.com/SongZeHNU/CFRN.

Open set recognition (OSR) aims to identify whether a test sample belongs to a semantic class in the classifier training set. Existing OSR methods exhibit prominent performance on various image datasets. However, they are primarily designed for general object recognition rather than more complex camouflaged object recognition. When an object is camouflaged, i.e., it exhibits a similar pattern to the background, it is difficult to finely identify it and differentiate between known and unknown categories. To address this problem, we propose a novel multi-granularity context perception network (MCPNet) for OSR of camouflaged objects, which can accurately identify camouflaged objects by fusing coarse-grained and fine-grained context features. In MCPNet, the vision transformer is first utilized to extract coarse-grained context features to locate the approximate location of camouflaged objects. Then, an adaptive local focus module (ALFM) is proposed to pick out the most discriminative regions and learn the fine-grained context of these regions. Finally, multi-granular context features are fused to obtain recognition results. During the training, a contrastive clustering module (CCM) is introduced to guide the network to effectively utilize multi-granularity context to generate high-confidence decision boundaries. We also built two camouflaged object classification datasets named ACOC and NCOC which mainly consist of artificial camouflage and natural camouflage respectively to facilitate research in OSR of camouflaged objects. Experimental results on two datasets show that MCPNet outperforms state-of-the art methods.

Spectral super-resolution aims to reconstruct a hyperspectral image (HSI) from its corresponding RGB image, which has drawn much more attention in remote sensing field. Recent advances in the application of deep learning models for spectral super-resolution have demonstrated great potential. However, these methods only work in spectral-spatial domain while rarely explore the potential property in the frequency domain. In this work, we first attempt to address spectral super-resolution in the frequency domain. To well merge the frequency information into the super-resolution network, a spectral-spatial-frequency domain fusion network (SSFDF) is designed, which consists of three key parts: frequency-domain feature learning, spectral-spatial domain feature learning, and feature fusion module. In more detail, a frequency-domain feature learning network is first exploited to dig the frequency-domain information of the input data. Then, a symmetric convolutional neural network (CNN) is developed to acquire the spectral-spatial features of the input data, where a parameter-sharing strategy is utilized to reduce network parameters. Finally, a feature fusion module is proposed to reconstruct HSI. Comprehensive experiments on several datasets reveal that our method can attain state-of-the-art reconstruction result with respect to other spectral super-resolution techniques.

Real-time semantic segmentation plays an important role in auto vehicles. However, most real-time small object segmentation methods fail to obtain satisfactory performance on small objects, such as cars and sign symbols, since the large objects usually tend to devote more to the segmentation result. To solve this issue, we propose an efficient and effective architecture, termed small objects segmentation network (SOSNet), to improve the segmentation performance of small objects. The SOSNet works from two perspectives: methodology and data. Specifically, with the former, we propose a dual-branch hierarchical decoder (DBHD) which is viewed as a small-object sensitive segmentation head. The DBHD consists of a top segmentation head that predicts whether the pixels belong to a small object class and a bottom one that estimates the pixel class. In this situation, the latent correlation among small objects can be fully explored. With the latter, we propose a small object example mining (SOEM) algorithm for balancing examples between small objects and large objects automatically. The core idea of the proposed SOEM is that most of the hard examples on small-object classes are reserved for training while most of the easy examples on large-object classes are banned. Experiments on three commonly used datasets show that the proposed SOSNet architecture greatly improves the accuracy compared to the existing real-time semantic segmentation methods while keeping efficiency. The code will be available at https://github.com/StuLiu/SOSNet.

Hyperspectral image (HSI) provides rich spatial and spectral information of ground objects, while synthetic aperture radar (SAR) records scattering information such as shape and structure. Fusion of HSI and SAR can improve the classification performance of land covers. In recent years, graph convolutional networks (GCN) have been widely used in the field of remote sensing due to its advantages in processing non-Euclidean structures, capturing local and global information. In this paper, we propose a spectral-spatial graph convolutional network (SSGCN) for fusion of HSI and SAR. First, the GCN is utilized to extract the spatial information of HSI and SAR. Then, a convolutional neural network is applied to extract the spectral information of HSI. Finally, the extracted spectral and spatial features are merged together followed by a fully connected layer to obtain the final classification result. Experiments on two datasets, i.e., Berlin and Augsburg, reveal that the proposed SSGCN significantly outperforms other representative methods.

Change detection (CD) in remote sensing imagery has become an increasingly substantial task in the monitoring of natural resources and disaster response. However, existing CD methods face a notable challenge in terms of resolution flexibility, wherein the resolution gap during training and testing stages results in a substantial performance decline. To address this issue, this paper first constructs a synthetic multiresolution benchmark through the resampling of existing public CD datasets. Subsequently, the impact of varying resolutions on CD models is systematically analyzed, accompanied by the introduction of a novel multi-resolution training (MRT) strategy aimed at improving the robustness of CD methods. Additionally, a plug-and-play module, termed the resolution pyramid constraint (RPC), is designed to facilitate alignments between multi-resolution features and original single-resolution features. Experiments demonstrate that small resampling factors could improve the performances of conventional CD methods, while large factors cause accuracies sharp decline. MRT and RPC are validated to be effective in enhancing robustness. Code and dataset will be available at https://github.com/flying318/MRCD.git.

Unsupervised domain adaption (UDA) has gained popularity in narrowing performance gaps across domains in remote sensing image semantic segmentation (RSISS). However, current UDA methods suffer from serious noisy pseudo-labels, adversely affecting domain adaptation performance. In this work, a local region homogenizing domain adaptation method (RegDA) is proposed to tackle this issue. Specifically, a generalized segment anything model (SAM) is utilized to obtain the semantic-consistent regions for the images in the target domain. Furthermore, a pixel-level voting scheme is proposed to get the semantic label for each local region and assign it to each pixel within this region. In this way, more reliable pseudo-labels are obtained and domain adaptation performance is improved. Experiment results on ISPRS datasets demonstrate that the proposed RegDA outperforms previous UDA approaches for RSISS. The code will be available at https://github.com/StuLiu/RegDA.

Hyperspectral change detection (HCD) aims to identify the changed and unchanged pixels in bitemporal images, which has been applied in various aspects. Currently, many deep learning-based change detection methods have been developed. However, existing change detection methods only focus on changed or temporal information while neglecting the complementary information between them. To solve this issue, a novel cross-temporal symmetric feature fusion network (CTSFFNet) is proposed for change detection of hyperspectral images. First, we perform pixel-wise subtraction and concatenation on the multi-temporal hyperspectral images to obtain the difference data and temporal data, respectively. Then, a three-layer convolutional neural network is performed on the difference data and temporal data to yield the difference and temporal features. Finally, a cross-temporal symmetric feature fusion (CTSSF) module is designed to merge the extracted features followed by a fully connected layer to obtain the final change regions. Experiments on two popular datasets demonstrate that the proposed CTSSFNet achieves superior detection performance compared to other state-of-the-art methods.

Unsupervised cross-scene hyperspectral image classification transfers the learnable knowledge from a labeled source scene to an unlabeled target scene. Currently, many statistical distribution alignment methods are introduced to mitigate domain discrepancy. However, these methods ignore the finer class specific structure which may cause negative transfer. To solve this issue, a prototype-based inter-intra domain alignment network is proposed for unsupervised cross-scene hyperspectral image classification. Specifically, a prototype-based inter-intra alignment method is proposed to narrow the feature distribution gap. Furthermore, an uncertainty estimation is developed to obtain highly reliable pseudo-labels in the target scene. Experiment results on several datasets imply that the proposed method outperform several cutting-edge unsupervised classification methods.

Remote sensing scene classification has garnered significant interest. However, few studies are dedicated to the classifi-cation of hyperspectral remote sensing scenes. Furthermore, current approaches that are designed for multispectral or visible images are not suitable for hyperspectral remote sensing scene classification. To solve these issues, a spectral-spatial convolutional network is proposed for hyperspectral remote sensing scene classification. First, a multiscale regional growth search method is developed to extract object region of the input data. Then, a three-branch network is designed to extract local spatial, global spatial, and spectral features. Finally, the obtained features are fed into fully connected layer to obtain class scores followed by a decision fusion rule to yield the final classification result. To evaluate the effect of hyperspectral remote sensing scene classification, a hyper-spectral remote sensing scene classification dataset (HRSS-C) is constructed, which consists of 1445 hyperspectral images with 11 different classes. Experiments on the HRSS-C reveal that our method gains superior classification performance with respect to other advanced approaches.

How to learn the most valuable and useful features in convolutional neural networks (CNNs) is the key for accurate hyperspectral image classification (HSIC). Focused on this issue, we developed a local–global gated CNN (LGG-CNN), in this letter. The core is the simultaneous construction of local and global gated convolution blocks, with the aim to select highly discriminative information and filtering redundant information in hyperspectral images (HSIs). Different from traditional CNN methods treating all spectral–spatial features equally, the gated convolutions help in learning a normalized soft mask to guide the network to focus on valid features and neglect the invalid ones. Here, based on the CNN backbone, multilayer local features are first learned via gated convolutional architecture, which mainly consists of convolution operators and nonlinearly activation functions. At the same time, a global gated block (GGB) is designed to conduct feature serialization-mapping-patching operations, to learn global features from deeper layers with larger receptive fields. As a result, the local/GGBs can dynamically learn discriminative feature selection mechanisms for each channel at each spatial location. Then, the local and global features are fused at both the feature-level and decision-level. In this manner, the effective fusion of features by the multilayer LGG convolution blocks enables spatial interaction across layers, leading to further improvement in classification accuracy. Extensive experiments on three benchmark HSIC datasets demonstrate the superiority of LGG-CNN over some state-of-the-art methods. The source code of the proposed method is available athttps://github.com/Ding-Kexin/LGG-CNN.

Visible-infrared vehicle target detection aims to pinpoint the location and class of vehicles by fusing the favorable complementary information of visible-infrared image pairs. However, most of detection methods cannot obtain ideal detection performance when visible-infrared image pairs are captured in low lighting environment. To solve this issue, we propose a global-local feature fusion network, which can adaptively integrate the saliency information from visible-infrared image pairs. Initially, a dual-stream ResNet-50 network is designed to extract cross-modal features from visible-infrared image pairs. Then, a global-local feature fusion module (GLF) is proposed to merge the multi-modality features. Finally, the detection head utilizes the fused features of the deep interaction to get the detection results. Experiments on the DroneVehicle and LLVIP datasets show that the proposed method is increased by 7.4% and 1.2 % compared to recently proposed methods, respectively.

Optical remote sensing is the major means of monitoring sea ice, which is beneficial for waterway planning, disaster prevention, and environmental research. Currently, a large number of sea ice segmentation methods have been developed. However, they often ignore the boundaries between sea ice and seawater, leading to unavoidable mis-segmentation. To alleviate this issue, this paper proposes a novel semantic segmentation network applied to sea ice remote sensing images, referred to as SegIceNet. First, we design a partial class activation map to select the feature points whose classes are easily confused in feature layers. By constructing the affinity among these points, the network explores potential semantic information and propagates these feature details among adjacent decoder layers. Second, this paper introduces a point-set optimization strategy, which aims to focus on samples of the aforementioned special points during the training process. Experimental results on a real sea ice dataset show that our method is better than other state-of-the-art semantic segmentation methods. Specifically, even in the imbalanced sea ice dataset, the proposed method achieved 94.88% mIoU.

Feature extraction is a prevalent technique in hyperspectral remote sensing. Various tasks require this technique as a pre-processing step, including image classification, anomaly detection, image denoising, and so on. Edge-preserving filtering based methods have been extensively utilized for this purpose. However, these methods do not take the inherent structural and textural information into account, leading to poor performance in classifying hyperspectral images (HSIs). In this letter, a new structural and textural-aware feature extraction method is proposed that preserves the relevant structural information and removes useless textures. First, structural and textural-aware recursive filtering features (STRFs) are extracted along with an exponential form of windowed inherent variance (eWIV). Then, multi-scale STRFs are integrated by the principal component analysis (PCA) method to obtain more discriminative features (MSTRF). Finally, the fused features are fed into a pixel-wise classifier to obtain the final results. The main difference between the MSTRF method and other feature extraction methods is that the MSTRF method can make full use of the proposed eWIV map, which can help to properly characterize structure and texture in HSIs. Experimental results on several public data sets indicate that our method leads to state-of-the-art classification performance, especially in the presence of very small training set.

Semi-supervised classification of remote sensing hyperspectral image (HSI) aims at exploiting both labeled and unlabeled samples for accurate land cover recognition. However, imbalanced data distribution and different classification difficulties negatively affect classification performance. Focused on this, a novel dual-stream class-adaptive network (DSCA-Net) is proposed for semi-supervised HSI classification, in this paper. First, a superpixel-guided label propagation module is introduced to alleviate the negative effect of imbalanced data distribution. Specifically, approximate estimation of labels for unlabeled samples is achieved via superpixel-wise similarity measure and label propagation, so that equal sampling is applied to each class. Then, a consistency regularization-based dual-stream network is constructed, which shares the same encoder for feature representation of either labeled or unlabeled samples. Based on this, two distinct classifiers are designed to force similar predictions can be achieved for various perturbed versions of the same unlabeled sample, thereby allowing unlabeled samples to train the model in a supervised manner. Finally, since different classes always have various degrees of learning difficulty, equal treatment may lead to overfitting of “easy” classes and biased prediction of “hard” classes. Unlike the traditional selection of unlabeled samples with a fixed threshold, dynamic class-adaptive thresholds are calculated according to the learning status of the model. In this manner, a higher threshold is assigned to “easy” classes to reduce sample redundancy, and a lower threshold is set for “hard” classes to select more samples. Experiment results demonstrate the effectiveness and superiority of the proposed method. Codes are available at https://github.com/luting-hnu/DSCA-Net.

Hyperspectral and synthetic aperture radar (SAR) image classification, aiming to merge multisource information to boost the precision and reliability of land cover classification, has gained increasing attention. Nevertheless, current techniques still exhibit certain limitations in extracting discriminative features and integrating heterogeneous features. In this work, a graph convolutional fusion network (GCFNet) is proposed for hyperspectral and SAR image classification. First, a spectral residual neural network is employed to extract the spectrum information. Then, a dual-branch graph convolutional network (GCN) is developed to extract the spatial information from hyperspectral and SAR images. Finally, a cross-contextual transformer fusion module is created to merge the spectral and spatial information followed by a dense layer to yield the final prediction outcome. To confirm the performance of the GCFNet, experiments on three datasets (e.g., Berlin, Augsburg, and Yellow River) demonstrate that the GCFNet significantly surpasses other representative methods.

In this article, a two-dimensional (2-D) spectral representation is proposed for the visualization and classification of hyperspectral images (HSIs). First, several sequence data processing methods, i.e., Gramian angular field (GAF) algorithm, Markov transition field (MTF), and recurrence plot (REP), are applied to obtain multiple 2-D features of a one-dimensional (1-D) spectrum. Second, the 2-D spectral features are stacked together to form the final 2-D spectral representation. Finally, many excellent classifiers in computer vision field are applied on the 2-D spectral representation to obtain the final classification result. Furthermore, 114 target spectral visualization maps are established based on their 1-D spectra. Experimental results reveal that the 2-D spectral representation has multiple advantages in terms of better visual quality and classification accuracies. The code of this work is available athttps://github.com/zhuyongxiang1/two-dimensional-spectral-representation.

Oriented object detection with remote sensing scenes has made excellent progress in recent years, especially using anchor-free detectors. Without the limitation of inherent prior spatial information, anchor-free detectors regress the detection boxes from the object center or edge in an elegant way. However, anchor-free detectors suffer severe feature misalignment and inconsistency between classification and regression. Especially in remote sensing scenes, there are densely arranged instances and multi-scale representations, which will affect the detection accuracy. Therefore, a feature augmentation module (FAM) and an oriented feature alignment (OFA) module are proposed for oriented object detection called FAA-Det. More specifically, we first introduce a FAM to enhance the object representation. After that, the augmented feature maps will be fed into OFA for feature alignment and accurate detection. OFA has two independent branches for classification and regression, and their separate structures can alleviate the inconsistency in detection. FAM and OFA comprise the FAA-Head in our detector. Extensive evaluation demonstrates the effectiveness of our proposed FAA-Det that performs the state-of-the-art (SOTA) mean average precision (mAP) on the DOTA and HRSC2016 datasets without bells and whistles. Our code will be available athttps://github.com/jimuIee/FAA-Det.

Domain adaptive segmentation has recently gained more and more attention in the remote sensing field. However, current methods often generate a significant number of uncertain examples, i.e., noisy pseudo-labels, in the target domain, which adversely affects model convergence. To solve this issue, an uncertain example mining network is proposed for domain adaptive segmentation of remote sensing images. Specifically, a novel strategy called multilevel pseudo-label correcting (MPC) is proposed to correct the pseudo-labels in class, pixel, and superpixel levels. In this way, more reliable pseudo-labels can be selected for the subsequent training stage. Furthermore, a noise-robust example mining strategy, termed uncertainty-based valuable example mining (UVEM), is proposed to prioritize confident examples with significant gradients for training effectively. Extensive empirical evaluations on IsprsDA and LoveDA datasets demonstrate that the proposed method outperforms previous approaches, establishing state-of-the-art results in domain adaptive remote sensing image segmentation (RSIS). The code will be available athttps://github.com/StuLiu/UemDA.

Remote sensing image change detection can analyze alterations on the Earth’s surface within a specific region. However, the accuracy of change detection has consistently been hindered by the style differences in captured images caused by seasonal or lighting variations, as well as the challenge of distinguishing similar features between the background and foreground in the scene. To this end, a multidomain differential excavating network (MDENet) for change detection is introduced. Using the novel multidomain differential collaboration module (MDCM) to precisely capture object features on the frequency and spatial domains across diverse temporal domains, it enables simultaneous querying of global and local change information. Moreover, the multineighborhood frequency gate attention (MFGatt) is devised to eliminate the impact of image style relevance information and consolidate attention toward object localization, thereby enhancing the adaptability of the network to variations in image style. Extensive experiments have illustrated that our proposed network achieves better detection accuracy compared with current state-of-the-art (SOTA) methods on various datasets.

Hyperspectral images (HSIs) offer detailed and abundant spectral-spatial information, holding great potential for ground object change detection (CD). Currently, numerous deep learning-based hyperspectral CD (HCD) models have been studied. However, these methods only consider single difference information or temporal information while neglecting the complementary advantage between difference and temporal information. To solve this issue, this work proposes a multitemporal symmetric fusion network (MTSFNet) for HCD, which involves three steps. First, the difference and temporal data are calculated by a subtraction operation and concatenation operation. Then, a dual-branch convolutional network is developed to capture the difference and temporal features. Finally, a symmetric feature fusion scheme is designed to integrate the extracted features followed by a fully connected layer to derive the detection results. Experiments conducted on multiple well-known datasets reveal that the proposed MTSFNet outperforms other advanced CD approaches in terms of qualitative and quantitative results.

Nowadays, high-resolution remote sensing images provide rich data sources and deep learning models show powerful feature representation capability for remote sensing object detection. However, due to the complex object structure as well as the changeable rotation angle, how to efficiently estimate the oriented bounding box regarding the accurate location of objects is still an open issue. Focused on this, a new one-stage structure-adaptive oriented object detection (SOOD) network is proposed, in this article. First, we designed a new rotation angle encoder (RAE), where an angle coordinate system is adopted and periodic angle correction is conducted. Different from the traditional longe-edge definition for angle estimation, the RAE can mitigate boundary discontinuity and square-like problems. Then, structure-adaptive label assignment (SALA) and confidence estimation (SACE) are introduced, to locate the position of objects more accurately. On the one hand, the anchor box determines the label assignment according to the affiliation relationship between the center point and the object’s inner ellipse boundary. By constraining the ellipse boundary and employing non-parametric label assignment, high-quality anchor boxes are initially selected, and low-quality anchor boxes are suppressed. On the other hand, the integration of intersection over union (IoU) prediction and uncertainty prediction constructs a quality evaluation function to guide. In this manner, this function dynamically evaluates the localization and classification ability of each prediction box. Extensive experiments on publicly available datasets such as DOTA1.0, DOTA1.5, DIOR, and MAR20 demonstrate the effectiveness of the proposed model. The source code will be available athttps://github.com/fan609/SOOD.

In the past few years, there has been significant progress in hyperspectral image classification (HSIC). However, when the trained classifier on the source scene is directly applied to a new scene, the classification performance tends to dramatically decrease because of the spectral shift phenomenon. Most existing techniques use feature alignment to learn knowledge from labeled scenes to unlabeled scenes, often overlooking the impact of noisy samples and outliers. To tackle this issue, the classwise prototype-guided alignment network (CPGAN) is proposed for cross-scene HSIC. The core idea is that classwise prototypes across scenes are employed as alignment intermediaries to guide cross-scene feature alignment. Specifically, first, spectral-spatial features from different scenes are extracted with a common feature extractor. Then, an uncertainty-aware pseudolabel selection (UPS) is designed to obtain high-confidence pseudolabels for unlabeled target scenes. Finally, a novel classwise prototype-guided alignment method is proposed to simultaneously achieve interdomain and intradomain alignment (IntraDA). The experimental results conducted on three datasets show that our method achieves superior performance compared to other cutting-edge classification algorithms.

Hyperspectral images (HSIs) have been widely used for target detection due to their abundant spatial and spectral information. In this article, a shadow-insensitive hyperspectral target detection (HTD) framework based on exposure fusion is proposed, which consists of the following major steps. First, the input HSI is divided into two parts, namely the shadow region and the nonshadow region. Second, total variation-based feature extraction and overexposure operation are performed on the input image to produce two feature images, i.e., the original feature image and the overexposure image. Third, a self-guided constrained energy minimization (SGCEM) detector is performed on the two feature images to detect the targets in shadow and nonshadow regions, respectively. Finally, the detection results obtained on the original feature image and the overexposure image are fused to acquire the final detection result. Extensive experiments conducted on real-world data illustrate that the proposed method can achieve satisfactory results when shadow exists.

Remote sensing super-resolution (SR) technique, which aims to generate high-resolution image with rich spatial details from its low-resolution counterpart, play a vital role in many applications. Recently, more and more studies attempt to explore the application of Transformer in remote sensing field. However, they suffer from the high computational burden and memory consumption for remote sensing super-resolution. In this paper, we propose an efficient Swin Transformer (ESTNet) via channel attention for SR of remote sensing images, which is composed of three components. First, a three-layer convolutional operation is utilized to extract shallow features of the input low-resolution image. Then, a residual group-wise attention module is proposed to extract the deep features, which contains an efficient channel attention block (ECAB) and a group-wise attention block (GAB). Finally, the extracted deep features are reconstructed to generate high-resolution remote sensing images. Extensive experimental results proclaim that the proposed ESTNet can obtain better super-resolution results with low computational burden. Compared to the recently proposed Transformer-based remote sensing super-resolution method, the number of parameters is reduced by 82.68% while the computational cost is reduced by 87.84%. The code of the proposed ESTNet will be available at https://github.com/PuhongDuan/ESTNet for reproducibility.

Image geo-localization aims to locate a query image from source platform (e.g., drones, street vehicle) by matching it with Geo-tagged reference images from the target platforms (e.g., different satellites). Achieving cross-modal or cross-view real-time (>30fps) image localization with the guaranteed accuracy in a unified framework remains a challenge due to the huge differences in modalities and views between the two platforms. In order to solve this problem, a novel fine-grained overlap estimation based image geo-localization method is proposed in this paper, the core of which is to estimate the salient and subtle overlapping regions in image pairs to ensure correct matching. Specifically, the high-level semantic features of input images are extracted by a deep convolutional neural network. Then, a novel overlap scanning module (OSM) is presented to mine the long-range spatial and channel dependencies of semantic features in various subspaces, thereby identifying fine-grained overlapping regions. Finally, we adopt the triplet ranking loss to guide the proposed network optimization so that the matching regions are as close as possible and the most mismatched regions are as far away as possible. To demonstrate the effectiveness of our FOENet, comprehensive experiments are conducted on three cross-view benchmarks and one cross-modal benchmark. Our FOENet yields better performance in various metrics and the recall accuracy at top 1 (R@1) is significantly improved, with a maximum improvement of 70.6%. In addition, the proposed model runs fast on a single RTX 6000, reaching real-time inference speed on all datasets, with the fastest being 82.3 FPS.

Camouflaged object detection (COD) aims to identify object pixels visually embedded in the background environment. Existing deep learning methods fail to utilize the context information around different pixels adequately and efficiently. In order to solve this problem, a novel pixel-centric context perception network (PCPNet) is proposed, the core of which is to customize the personalized context of each pixel based on the automatic estimation of its surroundings. Specifically, PCPNet first employs an elegant encoder equipped with the designed vital component generation (VCG) module to obtain a set of compact features rich in low-level spatial and high-level semantic information across multiple subspaces. Then, we present a parameter-free pixel importance estimation (PIE) function based on multiwindow information fusion. Object pixels with complex backgrounds will be assigned with higher PIE values. Subsequently, PIE is utilized to regularize the optimization loss. In this way, the network can pay more attention to those pixels with higher PIE values in the decoding stage. Finally, a local continuity refinement module (LCRM) is used to refine the detection results. Extensive experiments on four COD benchmarks, five salient object detection (SOD) benchmarks, and five polyp segmentation benchmarks demonstrate the superiority of PCPNet with respect to other state-of-the-art methods.

Hyperspectral image (HSI) classification methods have made great progress in recent years. However, most of these methods are rooted in the closed-set assumption that the class distribution in the training and testing stages is consistent, which cannot handle the unknown class in open-world scenes. In this work, we propose a feature consistency-based prototype network (FCPN) for open-set HSI classification, which is composed of three steps. First, a three-layer convolutional network is designed to extract the discriminative features, where a contrastive clustering module is introduced to enhance the discrimination. Then, the extracted features are used to construct a scalable prototype set. Finally, a prototype-guided open-set module (POSM) is proposed to identify the known samples and unknown samples. Extensive experiments reveal that our method achieves remarkable classification performance over other state-of-the-art classification techniques.

Hyperspectral change detection (HCD) is widely applied in various domains, such as accurate agriculture, disaster assessment, land use, and environmental monitoring. Most of hyperspectral change detection methods aim at extracting and classifying the spectral variation features with dimension reduction and machine learning methods. Different from previous work, this paper proposes an edge-guided hyperspectral change detection method. Specifically, a subtraction operation is adopted to extract difference hyperspectral image. Then, the edge-preserving filtering is performed on the difference HSI so as to extract spectral-spatial features. Next, the number of the extracted features is diminished through the kernel principal component analysis. Finally, the fused features are input into a spectral classifier followed by the edge-preserving filtering to obtain the final change detection result. Experiments on several HCD datasets demonstrate that the proposed method can consistently outperform other advanced approaches in both subjective and objective evaluations when only a limited number of labeled samples are available.

Weakly supervised learning using image-level annotations has become a popular choice for reducing labeling efforts of remote sensing object extraction. Existing methods exploit inter-pixel relations within an individual image patch for object localizations. When facing large-scale remote sensing images, it is still challenging to obtain global semantic contexts across image patches for feature representation, resulting in inaccurate object localizations. To remedy these issues, we propose a local-global anchor guidance network (LGAGNet) for weakly supervised landslide extraction. Specifically, a structure-aware object locating (SOL) module is developed to capture the spatial structure of landslide objects and extract local category anchors containing informative feature embeddings. Furthermore, we leverage a global anchor aggregation (GAA) module to excavate semantic patterns across image patches based on a memory bank, which is then used as additional context cues to enhance the feature presentation through a cross-attention mechanism. Finally, a hybrid loss function is designed to guide the network training, considering category-aware semantic contrasts and local activation consistency. Experimental results on high-resolution aerial and satellite image datasets verify the effectiveness of the proposed approach on landslide extraction.

Oil spill detection has attracted increasing attention in recent years since marine oil spill accidents severely affect environments, natural resources, and the lives of coastal inhabitants. Hyperspectral remote sensing images provide rich spectral information which is beneficial for the monitoring of oil spills in complex ocean scenarios. However, most of the existing approaches are based on supervised and semi-supervised frameworks to detect oil spills from hyperspectral images (HSIs), which require a massive amount of effort to annotate a certain number of high-quality training sets. In this study, we make the first attempt to develop an unsupervised oil spill detection method based on isolation forest for HSIs. First, a Gaussian statistical model is designed to remove the bands corrupted by severe noise. Then, kernel principal component analysis (KPCA) is employed to reduce the high dimensionality of the HSIs. Next, the probability of each pixel belonging to one of the classes of seawater and oil spills is estimated with the isolation forest, and a set of pseudo-labeled training samples is automatically produced using the clustering algorithm on the detected probability. Finally, an initial detection map can be obtained by performing the support vector machine (SVM) on the dimension-reduced data, and the initial detection result is further optimized with the extended random walker (ERW) model so as to improve the detection accuracy of oil spills. Experiments on hyperspectral oil spill data (HOSD) created by ourselves demonstrate that the proposed method obtains superior detection performance with respect to other state-of-the-art detection approaches. We will make HOSD and our developed library for oil spill detection publicly available at https://github.com/PuhongDuan/HOSD to further promote this research topic.

Hyperspectral oil spill mapping aims to distinguish the type of oil spill. Recently, most hyperspectral oil spill detection methods are based on supervised methods that work well with rich training samples. However, in the marine oil spill detection scenario, pixel annotations are difficult and costly. Moreover, the labels obtained by domain experts within a hyperspectral image (HSI) are often scarce. To address these issues, a self-supervised spectral-spatial transformer network is proposed for hyperspectral oil spill mapping. First, we propose a transformer-based contrastive learning network to extract the deep discriminative features. Then, the learned features are transferred to the downstream classification network that is fine-tuned with very few labeled samples. Experiments on hyperspectral oil spill database (HOSD) constructed by ourselves indicate that the proposed method can obtain more promising performance than several state-of-the-art oil spill classification techniques in discriminating different types of oil spills, i.e., thick oil, thin oil, sheen, and seawater.

Due to the impact of climate change, the frequency of flood events has increased in recent years, which puts forward an urgent need for timely and accurate flood mapping for emergency response. As the synthetic aperture radar (SAR) enables all-time monitoring regardless of bad weather conditions, it fits far better than passive optical sensors to delineate submerged areas during flood events. However, the universal, rapid, and accurate detection of flood extent remains a challenge. Drawing inspiration from the analysis of time-series variation in representative ground objects caused by flood events, as observed in a dual-polarized SAR time series over a hydrological year, we construct a novel window-based variation model. This model can be used to capture both long-term trends and short-term fluctuations of flood features across different polarization modes. Subsequently, we introduce an unsupervised flood mapping framework that integrates spatiotemporal flood features extracted by fuzzy-based methods. Given the distinct backscatter value of short vegetation, a flooded short vegetation activation model is designed and performed to enhance flood mapping accuracy in complex regions. The proposed method, tested on the 2020 East Dongting Lake flood in China, surpasses three unsupervised flood mapping methods and two deep learning methods in terms of quantitative evaluation and visual performance. The uncertainty of our proposed framework is tested through parameter sensitivity analyses, comparisons with flood mapping results from other sensor images, and extensive experiments on floods at different locations and times, thereby demonstrating its effectiveness, stability, and universality.

Convolutional neural networks (CNNs) have been widely employed for hyperspectral image (HSI) classification due to their powerful ability to extract local spatial features. However, CNN-based methods cannot establish long-range dependencies among sequences of pixels. Transformers offer significant advantages when processing sequential data and can establish global relationships, but they still encounter a number of challenges, such as their limited spatial feature extraction ability, or their high computational cost. In order to address the aforementioned issues, we develop a new fast HSI classification approach combining transformers and SimAM-based CNNs. The latter are utilized to extract better spatial features, where the complex spatial characteristics of HSIs are retrieved using an improved hierarchical 2D dense network structure. A dual attention unit (DAU) mechanism is then utilized to direct the model’s attention to discriminative spatial pixel characteristics and effective feature map channels, while suppressing information that is irrelevant for classification purposes. Regarding the spectral features, after extracting hierarchical local characteristics from various convolutional layers (using the hierarchical dense network structure), a squeezed-enhanced axial transformer is employed to establish global long-range dependencies whilst enhancing the ability of the model to extract local detail features in the HSI. Besides, a new Lion optimizer is utilized to improve the classification performance of our model. Our quantitative and comparative experiments on four benchmark datasets demonstrate the effectiveness of the proposed approach provides better classification results than other state-of-the-art approaches. Moreover, our FTSCN also achieves better classification results than other methods in practical scenarios.

In remote sensing image fusion, pansharpening is a type of remote sensing image fusion method that aims to fuse panchromatic (PAN) images and multispectral (MS) images to produce high-resolution multispectral (HRMS) images. Deep learning based pansharpening technology offers a series of advanced unsupervised algorithms. However, there are several challenges: (1) The existing unsupervised pansharpening methods only consider the fusion of single-scale features; (2) for the fusion of MS and PAN image feature branches, the existing pansharpening methods are implemented directly by concatenation and summation, without paying attention to critical features or suppressing redundant features; (3) the semantic gap in the long skip connections of the network architecture will create unexpected results. In this paper, we design a multiscale unsupervised architecture based on generative adversarial networks (GANs) for remote sensing image pansharpening (Mun-GAN), which consists of a generator and two discriminators. The generator includes a multi-scale feature extractor (MFE), a self-adaptation weighted fusion (SWF) module, and a nest feature aggregation (NFA) module. First, the MFE is utilized to extract multiscale feature information from the input images and to then pass this information to the SWF module for adaptive weight fusion. Then, multiscale features are reconstructed by the NFA module to obtain HRMS images. The two discriminators are spectral and spatial discriminators used against the generator. Moreover, we design a hybrid loss function to aggregate the multiscale spectral and spatial feature information. Compared with other state-of-the-art methods using QuickBird, GaoFen-2 and WorldView-3 images, which demonstrate that the Mun-GAN yields better fusion results.

Deep learning has been a powerful tool for hyperspectral image (HSI) classification. However, it is still an open issue to effectively learn highly discriminative features from the HSI, due to the high-dimensionality and complex spectral-spatial characteristics. To settle this issue, we propose a new band-grouping guided multi-attention module for the performance promotion of spectral-spatial feature learning. First, based on the fact of high relevance between adjacent spectral bands and low dependencies across long-range ones, all the spectral bands are adaptively divided into multiple non-overlapping groups where relevant bands are included. The advantage is to reduce the spectral dimension and data complexity when processing and analyzing each group. Then, a multi-attention mechanism, which not only explore the intra-group salient information but also propagate the inter-group difference information, is embedded into the convolutional neural networks to learn group-specific spectral-spatial features. By emphasizing useful spectral/spatial information and squeezing useless information with attention mechanism, the severability of learned features is enhanced. Based on this module, a spectral-spatial classification network is built, named by grouped multi-attention network (GMA-Net). The GMA-Net contains a two-branch architecture, i.e., pixel-wise spectral feature learning and patch-wise spectral-spatial feature learning. Via fusing the features from two branches, the complementary and discriminative features provided by pixel-wise and patch-wise learning manner can be integrated to further boost classification performance. Experimental results demonstrate that the proposed method is superior than several state-of-the-art approaches. Codes are available at: https://github.com/luting-hnu.

Recent studies have demonstrated the potential of hybrid convolutional models that combine 3D and 2D convolutional neural networks (CNNs) for hyperspectral image (HSI) classification. However, these models do not fully utilize the benefits of hybrid convolution due to inefficient connections between the two types of CNNs. Moreover, most CNNs, including hybrid models, require a significant number of parameters and computational resources for accurate classification, which increases the need for labeled samples and computational cost. Although the common lightweight strategies like depthwise separable convolution (DSC) can reduce parameters and computation compared to normal convolution (NC), they often compromise accuracy. To address these challenges, we propose a lightweight hybrid convolutional neural network (Lite-HCNet) for HSI classification with minimal model parameters and computational effort. Firstly, we design a novel channel attention module (NCAM) and combine it with a convolutional kernel decomposition (CKD) strategy to propose a lightweight and efficient DSC (LE-DSC) deployed in Lite-HCNet. The LE-DSC not only reduces the DSC volume further but also enhances its performance. Secondly, a lightweight and efficient hybrid convolutional layer (LE-HCL) is designed in Lite-HCNet to explore the efficient connection structure between 3D CNNs and 2D CNNs. Experiments show that the Lite-HCNet reduces the required computational cost and practical deployment difficulty while offering advanced performance with a small number of training samples. Furthermore, abundant ablation experiments confirm the superior performance of the designed LE-DSC.

Single-modal image registration methods are generally not feasible for visible and infrared images. Besides, multi-modal image registration methods still suffer from uneven distribution of extracted features, low repeatability, and ambiguous features. To address these issues, a coarse-to-fine infrared and visible image registration approach for dual sensor UAV imaging system is proposed, which is resilient to the difference of focal lengths and field of view. First, in the coarse registration step, the infrared image is transformed to the same scale as the visible image by using the similarity transformation. This operation makes the proposed method robust to the variation of field of view. Then, the feature point pairs are initialized using feature detectors in the infrared image’s blocked phase congruency feature map. Next, the feature point pairs are optimized by estimating the offset based on the relationship between the constructed feature descriptors. Finally, using elastic deformation, the pixel-level registered infrared image is obtained. Extensive experiments demonstrate the superior performance of the proposed coarse-to-fine image registration methodology in the real infrared-visible image pairs. The code and dataset are available at https://drive.google.com/drive/folders/1mpUWwHUbKTrBdOrNMNRRnuJclDUAC7nU?usp=sharing.

Change detection (CD) aims to find areas of specific changes in multi-temporal remote sensing images. The existing methods fail to adequately explore the cross-temporal global context, making the establishment of spatial-temporal deep global associations insufficient and inefficient. As a result, their performance is vulnerable to complex and various objects in changing scenes. Hence, we propose a cross-temporal context learning network, termed as CCLNet, where the intra- and inter-temporal long-range dependency are mined and interactively fused, to fully exploit the cross-temporal context information. Specifically, a lightweight convolutional neural network is first used to extract deep semantic features. Then, a well-designed cross-temporal fusion transformer (CFT) is proposed to locate the changing objects in the scene by establishing the long-range dependency across bitemporal images. Thanks to this, the temporal-specific information extraction and cross-temporal information integration are seamlessly integrated into the same network, thereby significantly improving the discriminative features of changing objects. Furthermore, this allows us using naive backbones with low computational cost to achieve reliable CD performance. Experiments on mainstream benchmarks show that our proposed method can handle CD task faster than state-of-the-art methods while maintaining better or comparable matching accuracy on a single RTX3090.

Pansharpening has benefited from the development of deep learning (DL) and has achieved excellent results. However, most DL-based methods extract local features by convolutional neural networks and do not integrate global features. Moreover, these methods only extract high-frequency features on the high-pass domain (HPD) or only consider image features on the intensity domain (ID). The method that only considers features in one domain may result in insufficient extraction of spatial and spectral features. Therefore, we propose a dynamic local–global network model on dual-domains, that is, HPD and ID. The dynamic local–global feature extraction block (DLGB) is designed to dynamically integrate local and global features to improve the representation capability of the network. To decrease the computational complexity of global feature extraction, a lightweight biaxial nonlocal attention (BNLA) that captures global spatial features in horizontal and vertical directions is proposed. Experiments on GeoEye-1, QuickBird, and WorldView-3 datasets show that the proposed method presents better fusion performance on objective evaluation indices and subjective perception.

With the improvement of imaging equipment, hyperspectral underwater target detection (HUTD) has raised much interest in recent years. The existing HUTD methods do not fully utilize spectral characteristics and need prior information about targets. Besides, the detection performance lacks verification in natural scenarios. In this paper, the authors propose a Feature Bands based Unsupervised underwater target Detection method (FBUD), which aims at finding the optimal feature bands to identify the underwater target near the coastline. Specifically, the normalized difference water index (NDWI) and unmixing technique are adopted to find the target and background pixels. Then, the spectral difference between the target and background is used to find the feature bands. With a simple and fast math operation of the feature bands, the probability map of the underwater target can be easily obtained. Besides, a new unmanned aerial vehicle (UAV)-borne hyperspectral image dataset named HNU-UTD is built for underwater target detection in real-world scenes. Experimental results obtained with the HNU-UTD dataset confirm the accuracy and effectiveness of the proposed detection method, which even outperforms supervised detection methods.

Deep convolutional neural networks (DCNNs) have become the leading tools for object extraction from very-high-resolution (VHR) remote sensing images. However, the label scarcity problem of local datasets hinders the prediction performances of DCNNs, and privacy concerns regarding remote sensing data often arise in the traditional deep learning schemes. To cope with these problems, we propose a novel federated learning scheme with prototype matching (FedPM) to collaboratively learn a richer DCNN model by leveraging remote sensing data distributed among multiple clients. This scheme conducts the federated optimization of DCNNs by aggregating clients’ knowledge in the gradient space without compromising data privacy. Specifically, the prototype matching method is developed to regularize the local training using prototypical representations while reducing the distribution divergence across heterogeneous image data. Furthermore, the derived deviations across local and global prototypes are applied to quantify the effects of local models on the decision boundary and optimize the global model updating via the attention-weighted aggregation scheme. Finally, the sparse ternary compression (STC) method is used to alleviate communication costs. Extensive experimental results derived from VHR aerial and satellite image datasets verify that the FedPM can dramatically improve the prediction performance of DCNNs on object extraction with lower communication costs. To the best of our knowledge, this is the first time that federated learning has been applied for remote sensing visual tasks.

Camouflaged object detection (COD) aims to discover objects that blend in with the background due to similar colors or textures, etc. Existing deep learning methods do not systematically illustrate the key tasks in COD, which seriously hinders the improvement of its performance. In this paper, we introduce the concept of focus areas that represent some regions containing discernable colors or textures, and develop a two-stage focus scanning network for camouflaged object detection. Specifically, a novel encoder-decoder module is first designed to determine a region where the focus areas may appear. In this process, a multi-layer Swin transformer is deployed to encode global context information between the object and the background, and a novel cross-connection decoder is proposed to fuse cross-layer textures or semantics. Then, we utilize the multi-scale dilated convolution to obtain discriminative features with different scales in focus areas. Meanwhile, the dynamic difficulty aware loss is designed to guide the network paying more attention to structural details. Extensive experimental results on the benchmarks, including CAMO, CHAMELEON, COD10K, and NC4K, illustrate that the proposed method performs favorably against other state-of-the-art methods.

Nowadays, high-resolution images with rich spectral information are necessary for earth observation. Remote-sensing image fusion is an effective method to provide high-resolution multispectral images, which are obtained by fusing high-resolution panchromatic images and low-resolution multispectral images. However, existing methods mostly use the same network for image feature extraction, without considering the differences among different pixels, resulting in that the extracted features are not accurate enough. This letter proposes a spatial dynamic selection network for remote-sensing image fusion. A dynamic feature extraction module composed of multiple spatial dynamic blocks (SDBs) and cross-scale context connection blocks (CSCBs) is designed. The SDB can extract image features according to the input by different networks, and realize dynamic selection of pixel features. Since the spatial structure and spectral characteristic of each pixel are different, two complementary branches are designed in the SDB to extract different features, which improves the capability of feature extraction. Multiscale network structure is designed to obtain more abundant information and the CSCB is used to integrate the information of different scales. Experimental results on GeoEye-1 and WorldView-3 datasets demonstrate the superiority of the proposed method.

Haze in hyperspectral images is a common phenomenon, which severely degrades the quality of the acquired data and limits its applications. In this letter, a dark-object subtraction model is proposed for haze removal of hyperspectral images, which mainly is composed of three steps. First, a haze density map is estimated according to the haze characteristic in different spectral channels. Then, we design a saliency measure method to automatically calculate haze abundance of different channels. Finally, the haze-free image is obtained through solving the dark-object subtraction model. Experiments on real and simulated datasets demonstrate that our method consistently outperforms other state-of-the-art dehazing techniques in terms of reconstructed performance and computational cost.

Semantic segmentation for remote sensing is a crucial but challenging task. Many supervised semantic segmentation methods rely heavily on a large-scale pixel-wise annotated data set, but it is time-consuming and laborious to provide manual annotation. However, due to the common domain shift of remote sensing images, a direct transfer might not perform well. Therefore, many unsupervised domain adaptation methods have been proposed to solve the data distribution discrepancy in remote-sensing data sets, but these methods cannot completely utilize the features extracted in the training process. In addition, the correlations between feature map channels are crucial for the pixel-wise classification task. In this letter, a covariance-based channel attention module is proposed to capture correlations by covariance metric and weighting the feature map channels. To further improve the domain adaptation performance, we propose a three-stage unsupervised domain adaptation semantic segmentation method for remote-sensing images, we fine-tune the model which has been trained on the source domain on the target domain via self training and knowledge distillation. To test the effectiveness of the proposed method, experiments are conducted on the ISPRS 2-D Semantic Labeling data set and an urban drone data set. Our method shows a better performance advantage compared with other state-of-the-art methods.

Shadow removal is a challenging problem in hyperspectral remote sensing images due to its spatial-variant properties and diverse patterns. In this work, a shadow removal framework with multiexposure fusion is proposed for hyperspectral remote sensing images, which consists of three major steps. First, a color space conversion method is exploited to detect the shadow regions. Second, the principle of the intrinsic decomposition model is utilized to generate a set of differently exposed hyperspectral images (HSIs), i.e., multiexposure images. Third, the generated multiexposure images and the original HSIs are fused together with a two-stage image fusion method so as to remove the shadows in hyperspectral remote sensing images effectively. Experiments performed on three real hyperspectral datasets confirm that the performance of the proposed method outperforms other state-of-the-art shadow removal approaches.

Exploring fast and effective spectral-spatial feature extraction algorithms for hyperspectral image (HSI) classification is one of the most focus problems in current hyperspectral remote-sensing research. Generally, the size of homogeneous regions in HSIs is not consistent in real scenario and real scenario usually consist of ground objects of different scales. Multiscale strategy starts to be used to construct discriminative features at different scales for HSI classification in recent years. To efficiently characterize the multiscale spectral-spatial features of HSIs, a curvature filters-based multiscale feature extraction method with multiscale superpixel segmentation constraint is proposed. The proposed algorithm is composed of the following major stages. First, global multiscale spectral-spatial features are efficiently extracted via progressively curvature filtering and downsampling operations, which can be regarded as an image pyramid decomposition method. Next, a multiscale superpixel segmentation strategy is applied on the first layer of the image pyramid, and a weighted mean operation is applied within and among superpixels to extract the local multiscale spatial features (LMSFs). Finally, the global multiscale curvature features (GMCFs) and the superpixel segmentation-based LMSFs are fused to form the final multiscale spectral-spatial features for classification purposes. To verify the capabilities of the proposed method, comprehensive experiments are performed on five real hyperspectral datasets. Experimental results demonstrate that the proposed method can significantly improve the classification accuracies compared to several standard HSI feature extraction and classification methods, especially when the number of samples for training is limited.

Deep convolutional neural networks (CNNs) have made great progress in the super-resolution (SR) of hyperspectral images (HSIs). However, most methods utilize convolution to explore local features, and global features are ignored. It is expected that combining non-local mechanism with CNN will improve the performance of HSI SR. This paper presents a multi-level progressive HSI SR network. The dense non-local and local block (DNLB) is constructed to combine local and global features, which are used to reconstruct super-resolution images at each level. Due to the high dimension of HSI, original non-local methods produce memory-expensive attention maps. We develop a non-local channel attention block to extract the global features of HSIs efficiently. Spatial-spectral gradient is injected in the non-local attention block to obtain better details. Furthermore, the progressive learning mode based multi-level network is proposed to reconstruct HSI with fine details. A number of experiments demonstrate that our method can reconstruct hyperspectral images more accurately than existing methods.

Fog in hyperspectral images severely limits the visibility of imaging scene and reduces the image contrast, which has a negative effect on the following image interpretation. Defogging methods aim at restoring a high-quality image from the degraded image. Currently, most dehazing methods mainly depend on the atmospheric scattering model in computer vision and multispectral image communities. However, when these approaches are directly used to remove the fog from HSIs, they cannot produce satisfactory defogging performance. To alleviate this issue, we develop a novel fog model to achieve fog removal from hyperspectral images. First, a fog density map is calculated by differentiating the averaged bands falling into visible and infrared spectral ranges. Then, haze abundance in different spectral bands is estimated based on the pixel reflectance between two selected pixels with different haze levels. Finally, the high-quality hyperspectral image is restored by solving the defogging model. Experiments performed on a new benchmark created by ourselves demonstrate that the proposed method obtains favorable dehazing performance in contrast to other approaches in computer vision and remote sensing fields.

Remote sensing image super-resolution aims to boost the image resolution while recovering rich high-frequency details. Currently, most of super-resolution methods are based on an assumption that the degradation kernel is a specific downsampler. However, the degradation kernel is unknown and sophisticated for real remote sensing scenes, leading to a severe performance drop. To alleviate this problem, we propose a multi-layer degradation representation-guided blind super-resolution method for remote sensing images, which mainly consists of three key steps. First, an unsupervised representation learning is exploited to learn the degradation representation from low-resolution images. Then, a degradation-guided deep residual module is designed to model high-order features across different scales from original images. Finally, a multi-layer degradation-aware feature fusion mechanism is proposed to restore the finer details. Experiments on synthetic and real datasets demonstrate that the proposed method can achieve promising performance with respect to other state-of-the-art super-resolution approaches.

Oil spill mapping is a very challenging problem in marine environmental monitoring. In this paper, the potential of hyperspectral image classification for mapping oil spills is comprehensively investigated. First, several representative hyperspectral image classification methods are reviewed in a general framework. Second, three oil spill mapping cases are designed to analyze the performance of different classification methods in detecting the spatial distribution, classifying the type, and estimating the thickness of oil spills. Finally, the experimental results are analyzed in detail, and some conclusions are given, which bring a comprehensive understanding to scholars who are interested in the fields of hyperspectral remote sensing and oil spill mapping.

Semisupervised semantic segmentation is an effective way to reduce the expensive manual annotation cost and take advantage of the unlabeled data for remote sensing (RS) image interpretation. Recent related research has mainly adopted two strategies: self-training and consistency regularization. Self-training tries to acquire accurate pseudo-labels to explicitly expand the train set. However, the existing methods cannot accurately identify false pseudo-labels, suffering from their negative impact on model optimization. The consistency regularization constrains the model by producing consistent predictions robust to the perturbations introduced in the sample or feature domain but requires a sufficient number of training data. Therefore, we propose a strategy for the semisupervised semantic segmentation of the RS images. The proposed model in the generative adversarial network (GAN) framework is optimized by consistency self-training, learning the distributions of both labeled and unlabeled data. The discriminator is optimized by accurate pixel-level training labels instead of the image-level ones, thereby assessing the confidence for the prediction of each pixel, which is then used to reweight the loss of the unlabeled data in self-training. The generator is optimized with the consistency constraint with respect to all random perturbations on the unlabeled data, which increases the sample diversity and prompts the model to learn the underlying distribution of the unlabeled data. Experimental results on the the large-scale and densely annotated Instance Segmentation in Aerial Images Dataset (iSAID) datasets and the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets show that our framework outperforms several state-of-the-art semisupervised semantic segmentation methods.

Due to rich spectral information, hyperspectral images (HSIs) have been widely used in various fields. However, limited by imaging systems, the low spatial resolution of HSIs has become an important problem. In this article, for enhancing the spatial resolution, Interactformer is proposed to interact with global and local features extracted by Transformer and 3D convolutional neural network (CNN) branches. Within the Transformer branch, a separable self-attention module with linear complexity is designed to solve the problem that traditional self-attention mechanisms suffer from large memory costs due to quadratic complexity. In the 3D CNN branch, the spectral attention module and 3D convolution are applied jointly to better protect the spectral correlation among spectral bands and facilitate local feature extraction of HSIs. The interactive attention unit between the two parallel branches is designed to interact with local and global feature information adaptively. Compared with state-of-the-art super-resolution (SR) methods, the proposed method reconstructs better HSI in simulated SR experiments, real SR experiments, and classification experiments, which prove that Interactformer can effectively improve the spatial resolution while preserving the spectral information.

Unmanned aerial vehicle (UAV) hyperspectral imaging has been extensively applied in various fields. However, due to the limited imaging width, hyperspectral images (HSIs) captured by UAV need to be stitched, so as to effectively cover the study area. In this article, an effective seamless stitching method with deep feature matching and elastic warp is proposed for HSIs, which consists of the following major steps. First, for each input HSI, a single-band gray-scale image is obtained by fusing the bands corresponding to the red, green, and blue wavelengths. Second, the feature points of each HSI are obtained with a robust VGG-style network and matched with a graph neural network. After point pairs are obtained, the next step is to estimate the transformation matrix of adjacent images, and a spectral correction method based on intrinsic decomposition is proposed to ensure the spectral consistency of adjacent images. In the final stage, a seam-cutting and multiscale blending strategy is adopted to ensure the spatial consistency of the stitching results. Experimental results on real HSIs show that the proposed method is superior to six representative image stitching approaches.

Coastal wetland monitoring plays an important role in the protection and restoration of ecosystems in this world. UAV-hyperspectral imaging, as an emerging technique for Earth observation and space exploration, provides the huge potential ability to identify different wetland species. In this work, a multilayer global spectral–spatial attention network (MGSSAN) is proposed for mapping coastal wetlands, which mainly consists of two major steps. First, a two-branch convolutional neural network (CNN) framework with residual connection is developed to obtain an initial classification probability map, in which one branch is used to capture the spectral information, the other branch is used to extract spatial information, and a global spectral–spatial attention module is designed to guide networks focusing on those features that are more discriminative. Second, an extended random walker method is utilized to optimize the initial classification probabilities, so as to yield the final map. Experiments performed on three wetland HSI datasets created by ourselves verify that the proposed method can obtain superior performance with respect to several state-of-the-art hyperspectral image classification methods.

Feature extraction provides an effective tool to classify hyperspectral images (HSIs). However, most hyperspectral feature extraction methods tend to yield an over-smoothed phenomenon, which leads to inconsistency between the homogeneous regions and the ground objects in the actual scene. To alleviate this problem, an edge-aware feature extractor called contour structural profiles (CSPs) is proposed to extract the discriminative features for hyperspectral images classification (HSIC). The proposed classification method comprises three components. First, the spectral dimension of the HSI is reduced with an averaging-based method. Then, an edge-aware total variation (TV) model is constructed to extract the contour structural profile, in which a learned contour probability map is served as one of the major cues in the feature extraction process. Next, multiscale structural profiles (MSSPs) are constructed using the edge-aware TV model with different parameters so as to fully characterize ground objects with different scales. Finally, the MSSPs are fused with a kernel principal component analysis (KPCA) followed by a spectral classifier to obtain the final classification map. Experimental results on several publicly available hyperspectral datasets illustrate that the proposed method obtains superior classification performance over several state-of-the-art classification approaches, especially when the number of training samples is insufficient.

Combining spectral and spatial information can significantly improve the classification performance of hyperspectral image (HSI). Currently, a lot of spectral–spatial HSI classification methods have been proposed. However, the task of HSI classification has remained challenging since the number of training samples is limited in real scenarios. In this article, we propose a novel HSI classification framework with single sample, in which the spectral self-similarity and spatial polygon structure information are fully combined to improve the classification performance. On the one hand, spectral self-similarity is used to expand training samples, which makes it possible to obtain sufficient samples with minimal cost. On the other hand, polygonal partition is introduced to acquire the geometrical structure of land covers in man-made environments. Specifically, the edge information of geometric objects is captured by polygonal partition, which can be utilized to constrain the spatial range of sample expansion and optimize the classification results. Experimental results on three real HSIs illustrate that the proposed method performs very well under small training sample size even when the number of samples is single per class.

Manual and automatic annotation of Remote Sensing (RS) scenes are rather complex tasks which may unavoidably introduce some degree of mislabeled data in large-scale archives. In this regard, noisy annotations become an important constraint for deep metric learning-based RS characterization methods since most of them are trained in a supervised way. To address this problem, here we investigate the use of deep metric learning for characterizing RS scenes with noisy labels. Specifically, we consider the Normalized Softmax Loss and develop a robust extension, i.e., the Robust Normalized Softmax Loss (RNSL), in order to effectively capture the semantic relationships among RS scenes with mislabeled ground-truth information. The conducted experiments, using the K-NN classifier and two benchmark RS image archives, show the potential of the proposed approach with respect to other state-of-the-art methods.

Hyperspectral image classification has been a constant hot topic in remote sensing field, and achieved significant progress recently. Until now, most of the existing works are based on high-quality noise-free datasets, whereas in real applications, the images are often degraded by different types of noise, which makes the noise sensitivity become one of the key issues for classification assessment. In this paper, we study the noise effects on hyperspectral image classification including Guassian, salt-and-pepper, and stripe noise. The experimental results shows that noise has varying degrees of negative effects on different hyperspectral classification methods, which provides instructional information for method design and selection under noise environment in actual classification applications.

Fusion of hyperspectral images (HSIs) and multispectral images (MSIs) with different resolutions is an active research topic in the field of remote sensing. However, HSI-MSI fusion assessments in existing researches are basically conducted on simulated data gone through spectral or spatial downsampling, which cannot reflect the actual performances of fusion methods in application scenarios. To conquer this problem, a new remote sensing dataset- Hunan UAV-borne HSIs and MSIs fusion(HNU-HMiF) dataset is provided in this paper. The proposed dataset contains fine registered hyperspectral and multispectral image pairs captured by unmanned aerial vehicle (UAV) covering different ground objects, and can be used to evaluate, select, and even develop fusion methods for users or researchers. Successful applications including method evaluation and comparison confirm the validity and reliability of the proposed dataset.

In this paper, a novel unmanned aerial vehicle (UAV) hyperspectral image stitching framework based on radiation correction and seam-cutting blending is proposed. Firstly, spectral correlation constraints are introduced to eliminate mismatched pairs in the transform matrix estimation step. Then, a spectral correction method based on intrinsic images is proposed to ensure spectral consistency of stitching results. In order to obtain more natural stitching results without edge effect, a seam-cutting and multi-scale blending strategy is adopted in the final blending stage. Experimental results on real unmanned aerial vehicle hyperspectral strip images show that the proposed method is superior to a representative image stitching approach.

In this paper, a novel edge guided structure extraction method is proposed for hyperspectral images classification, which consists of the following steps: First, the spectral dimension of the hyperspectral image is reduced with an averaging-based method. Then, the structural features is extracted by an extended relative total variation (ERTV) inspired by a learned edge probability map which serves as one of the major cues in the structure extraction process. Finally, the extracted structural features are fed into SVM for classification. Experimental results on two publicly available hyperspectral data sets demonstrate the competitive performance over several state-of-the-art classification approaches.

It is well known that classification accuracy highly relies on the number of labeled samples. However, it is difficult to obtain sufficient labeled samples in real-world applications. To solve this issue, a novel hyperspectral image (HSI) classification method based on polygonal partition is proposed for crop mapping. This method only needs single sample per class as an initial training set. Specifically, multiscale polygonal partition is applied on the first three components of the HSI. Then, a spectral similarity-based sample expansion method is proposed to obtain more labeled samples. Next, a pixel-wise classifier, the support vector machine (SVM), is used to acquire an initial classification result. Finally, classification result is further optimized according to the partition maps. Experimental results show that classification performance of the proposed method is satisfactory even when the number of labeled sample is single for each class.

The inclusion of spatial information into spectral classifiers for fine-resolution hyperspectral imagery has led to significant improvements in terms of classification performance. The task of spectral-spatial hyperspectral image (HSI) classification has remained challenging because of high intraclass spectrum variability and low interclass spectral variability. This fact has made the extraction of spatial information highly active. In this work, a novel HSI classification framework using the fusion of dual spatial information is proposed, in which the dual spatial information is built by both exploiting pre-processing feature extraction and post-processing spatial optimization. In the feature extraction stage, an adaptive texture smoothing method is proposed to construct the structural profile (SP), which makes it possible to precisely extract discriminative features from HSIs. The SP extraction method is used here for the first time in the remote sensing community. Then, the extracted SP is fed into a spectral classifier. In the spatial optimization stage, a pixel-level classifier is used to obtain the class probability followed by an extended random walker-based spatial optimization technique. Finally, a decision fusion rule is utilized to fuse the class probabilities obtained by the two different stages. Experiments performed on three data sets from different scenes illustrate that the proposed method can outperform other state-of-the-art classification techniques. In addition, the proposed feature extraction method, i.e., SP, can effectively improve the discrimination between different land covers.

Pan-sharpening is an effective method to obtain high-resolution multispectral images by fusing panchromatic (PAN) images with fine spatial structure and low-resolution multispectral images with rich spectral information. In this article, a multiscale pan-sharpening method based on dynamic convolutional neural network is proposed. The filters in dynamic convolution are generated dynamically and locally by the filter generation network which is different from the standard convolution and strengthens the adaptivity of the network. The dynamic filters are adaptively changed according to the input images. The proposed multiscale dynamic convolutions extract detail feature of PAN image at different scales. Multiscale network structure is beneficial to obtain effective detail features. The weights obtained by the weight generation network are used to adjust the relationship among the detail features in each scale. The GeoEye-1, QuickBird, and WorldView-3 data are used to evaluate the performance of the proposed method. Compared with the widely used state-of-the-art pan-sharpening approaches, the experimental results demonstrate the superiority of the proposed method in terms of both objective quality indexes and visual performance.

Most deep metric learning-based image characterization methods exploit supervised information to model the semantic relations among the remote sensing (RS) scenes. Nonetheless, the unprecedented availability of large-scale RS data makes the annotation of such images very challenging, requiring automated supportive processes. Whether the annotation is assisted by aggregation or crowd-sourcing, the RS large-variance problem, together with other important factors [e.g., geo-location/registration errors, land-cover changes, even low-quality Volunteered Geographic Information (VGI), etc.] often introduce the so-called label noise, i.e., semantic annotation errors. In this article, we first investigate the deep metric learning-based characterization of RS images with label noise and propose a novel loss formulation, named robust normalized softmax loss (RNSL), for robustly learning the metrics among RS scenes. Specifically, our RNSL improves the robustness of the normalized softmax loss (NSL), commonly utilized for deep metric learning, by replacing its logarithmic function with the negative Box–Cox transformation in order to down-weight the contributions from noisy images on the learning of the corresponding class prototypes. Moreover, by truncating the loss with a certain threshold, we also propose a truncated robust normalized softmax loss (t-RNSL) which can further enforce the learning of class prototypes based on the image features with high similarities between them, so that the intraclass features can be well grouped and interclass features can be well separated. Our experiments, conducted on two benchmark RS data sets, validate the effectiveness of the proposed approach with respect to different state-of-the-art methods in three different downstream applications (classification, clustering, and retrieval). The codes of this article will be publicly available fromhttps://github.com/jiankang1991.

Hyperspectral image (HSI) and multispectral image (MSI) fusion, which fuses a low-spatial-resolution HSI (LR-HSI) with a higher resolution multispectral image (MSI), has become a common scheme to obtain high-resolution HSI (HR-HSI). This article presents a novel HSI and MSI fusion method (called as CNN-Fus), which is based on the subspace representation and convolutional neural network (CNN) denoiser, i.e., a well-trained CNN for gray image denoising. Our method only needs to train the CNN on the more accessible gray images and can be directly used for any HSI and MSI data sets without retraining. First, to exploit the high correlations among the spectral bands, we approximate the desired HR-HSI with the low-dimensional subspace multiplied by the coefficients, which can not only speed up the algorithm but also lead to more accurate recovery. Since the spectral information mainly exists in the LR-HSI, we learn the subspace from it via singular value decomposition. Due to the powerful learning performance and high speed of CNN, we use the well-trained CNN for gray image denoising to regularize the estimation of coefficients. Specifically, we plug the CNN denoiser into the alternating direction method of multipliers (ADMM) algorithm to estimate the coefficients. Experiments demonstrate that our method has superior performance over the state-of-the-art fusion methods.

Sun glint, as the spectral reflection of solar radiation on non-flat water surfaces, is a serious confounding factor for coastal shallow-water environments. When the coastal areas are observed with a hyperspectral sensor, the existing sun glint in the produced images can seriously influence the quality of the image interpretation. To solve this issue, in this paper, we propose a novel sun glint removal method based on a variation model for hyperspectral images (HSIs). The proposed method aims to decompose the original HSI into a desired clean image and a sun glint image. To achieve this, we exploit a texture-aware total variation to remove the sun glint in HSIs, where the texture information is imposed on the total variation regularization to highlight sun glint. Experiments on simulated and real datasets demonstrate that our method can obtain outstanding performance with respect to other state-of-the-art approaches.

Hyperspectral imaging plays an important role for mineral mapping in a nondestructive and noninvasive way. In this paper, a novel resolution enhancement method is proposed based on the principle of intrinsic image decomposition for mineral mapping. This method is based on an assumption that hyperspectral image (HSI) can be decomposed into a reflectance component and an illumination component. Based on this idea, the RGB image is first transformed into Intensity-Hue-Saturation (IHS) space, and the intensity channel is considered as the illumination component of the HSI with an ideal high spatial resolution. Then, the reflectance component of the ideal HSI is estimated with the downsampled HSI image and the downsampled intensity channel. Finally, the HSI with high resolution can be reconstructed by utilizing the estimated illumination and the reflectance components. Experimental results validate the effectiveness of the proposed method qualitatively and quantitatively by outperforming several state-of-the-art approaches.

In this paper, in order to extract efficient spectral-spatial features for hyperspectral image classification, a Gaussian curvature (GC) filter based feature extraction method with multiscale segmentation constraint is proposed. The method consists of the following major steps: First, the maximum noise fraction (MNF) method is applied on the hyperspectral images (HSIs) to reduce the noise and computational complexity. The GC features are extracted from the dimension reduced HSIs via the GC filter. Next, a multiscale segmentation strategy is applied on the HSIs, and the multiscale spatial features are extracted by applying the weighted mean operations within and among superpixels. Finally, the GC features and dimension reduced multiscale spatial features are fused to form the final multiscale Gaussian curvature features (MGCFs) for classification purposes. To verify the effectiveness of the proposed method, we conduct experiments on the Indian Pines data set. Experimental results demonstrate that the proposed method can significantly improve the classification accuracies compared to several standard classification methods.

Noise usually appears in hyperspectral images (HSIs), and strongly affects the performance of the follow processing and analysis. In recent years, a large number of denoising algorithms have been proposed and it is known that the denoising effect is highly dependent on the accurate estimates of the type and level of noise present in an HSI. This paper focuses on analyzing the real noise in HSIs by separating and estimating the level of noise in HSIs. In consideration of the spectral correlation and the unique spatial structure of stripe noise, the developed method employs Fourier domain analysis and the high correlation among neighboring spectral bands to separate different types of noise. Experimental results show that the level of noise may be quite different for different bands of an HSI, and HSIs captured by different senors or in different scenes.

Hyperspectral Image (HSI) visualization, which aims at displaying as much material information of original images as possible on a trichromatic monitor with natural color, plays an important role in image interpretation and analysis. However, most of the HSI visualization methods only focus on presenting the detail information of a scene without providing natural colors and distinguishing land covers with similar colors. In order to address this problem, this article proposes a multichannel pulse-coupled neural network (MPCNN)-based HSI visualization method, which consists of the following steps. First, the MPCNN is proposed and explored to fuse the original HSI so as to obtain a fused band with rich spatial details. Then, a color mapping scheme is proposed to determine the weights of red, green, and blue (RGB) channels. Finally, the weighted RGB channels are stacked together for visualization. Experiments performed on four hyperspectral data sets demonstrate that the proposed method not only displays the HSI with nature colors but also improves the details in the image. The effectiveness of the proposed method is demonstrated in terms of both visual effect and objective indexes.

The quantity and quality of training samples have a great influence on the performance of most hyperspectral image classification approaches. However, in a real scenario, manually annotating a large number of accurate training samples is extremely labor-intensive and time-consuming. In this article, a multilabel training sample augmentation method is proposed. Instead of giving an exact label to each pixel, we just precisely label a small number of pixels by giving them a single label (called single-label samples) and annotate a large number of pixels in certain regions together by giving them multiple labels (called multilabel samples). Furthermore, in order to make full use of the multilabel training samples, a superpixel segmentation and recursive filtering-based method is proposed. The proposed method consists of the following major steps: recursive filtering-based feature extraction, superpixel-based segmentation, and spectral-spatial similarity-based mislabeled sample removal. Experimental results demonstrate that the proposed method can significantly improve the classification accuracy of multiple classifiers by using the multilabel training samples.

In this article, a novel game theory-based approach is proposed for anomaly detection in hyperspectral images (HSIs) via effectively exploring multiple spectral and spatial characteristics of anomalies. This approach comprises three main steps. First, spectral, extend morphological profiles (EMPs), and Gabor texture features are captured from an input HSI. Then, we define the anomaly detection problem as an anomaly game model, in which image regions (superpixels) of different features are modeled as players who select to be “anomaly” or “background” as their strategies. Three initial detection results are produced based on each player's strategy in the Nash equilibrium of the anomaly game. Last, a saliency-based decision fusion technique is used to combine the complementary information in different features, so as to obtain a fused detection map. The performance of the proposed anomaly detection technique is evaluated on four real-scene HSIs. Experimental results validate that our approach can outperform some state-of-the-art anomaly detection methods.

In this article, a novel hyperspectral anomaly detection method with kernel Isolation Forest (iForest) is proposed. The method is based on an assumption that anomalies rather than background can be more susceptible to isolation in the kernel space. Based on this idea, the proposed method detects anomalies as follows. First, the hyperspectral data are mapped into the kernel space, and the first K principal components are used. Then, the isolation samples in the image are detected with the iForest constructed using randomly selected samples in the principal components. Finally, the initial anomaly detection map is iteratively refined with locally constructed iForest in connected regions with large areas. Experimental results on several real hyperspectral data sets demonstrate that the proposed method outperforms other state-of-the-art methods.

In this paper, a novel deep learning based visualization method is proposed for natural color visualization of hyperspectral images, which consists of the following steps. First, the spectral bands of the hyperspectral image are divided into two groups, i.e., the red, green, and blue (RGB) bands and the remaining bands. Then, a pretrained convolutional neural network (CNN) model, i.e., VGG-19, is explored to fuse the remaining bands so as to obtain a fused band with rich details. Next, the intensity-hue-saturation (IHS) transform is performed on the averaged red, green, and blue bands to obtain three different components, i.e., intensity (I), hue (H), saturation (S). Finally, the fused band is utilized to replace the intensity component followed by an inverse IHS transform. Experiments performed on two hyperspectral data sets demonstrate that the proposed method cannot only obtain a natural color resulting image, but also well preserving image details with respect to several state-of-the-art methods.

The number and quality of training samples have a big influence on hyperspectral image classification performance. However, it is often difficult to manually annotate a large number of accurate training samples because the annotation requires a lot of manpower and resources. In this paper, we first propose a multi-labeling method to label the training samples efficiently. Instead of giving the exact label for each training pixel, we just precisely label a small number of pixels (called single-label samples), and annotate a large number of pixels in certain regions together (called multi-label samples) with multiple labels. Furthermore, a superpixel segmentation and recursive filtering based sample enhancing method is proposed to make full use of multi-label training samples for classification, which consists of the following major steps: IFRF based feature extraction, superpixels based classification, and spatial-spectral similarity based inaccurate samples removal. Experimental results demonstrate that the proposed method can improve the classification accuracy of multiple classifiers with multi-label training samples.

A novel anomaly detection method based on Isolation Forest is proposed for hyperspectral images. The method is directly based on a concept that anomalies rather than background can be more susceptible to isolation in the feature space. Based on this idea, the proposed method detects anomalies by directly isolating anomaly pixels from background. First, the probabilities of the anomalies in HSI is first evaluated by global Isolation Forest to obtain an initial anomaly detection map. Then, the initial anomaly detection map is recursively refined with local Isolation Forest so as to make full use of the spatial information. Experimental results on two real hyperspectral data sets demonstrate that the proposed method is superior over several widely used anomaly detection approaches.

In this letter, a coarse-to-fine unsupervised method is proposed for cloud detection in remote sensing images. First, the color, texture, and statistical features of the remote sensing images are extracted with the color transform, dark channel estimation, Gabor filtering, and local statistical analysis methods. Then, an initial cloud detection map can be obtained by performing the support vector machines (SVM) on the stacked features, in which the SVM is trained with a set of samples automatically labeled by processing the dark channel of the original image with several thresholding and morphological operations. Finally, guided filtering is used to refine the boundaries in the initial detection map, which further improves the cloud detection accuracy. Experiments performed on several real remote sensing images demonstrate that the proposed method show better detection performances with respect to several recently proposed cloud detection methods in terms of both quantitative and visual comparisons.

Recently, convolutional neural networks (CNNs) as a powerful tool have been introduced for classification of hyperspectral images (HSIs). However, it fails to take the feature redundancy into consideration. Hence, for the pixel-wise HSI classification, the CNN-based methods may not effectively extract the discriminative features from the complex scene in HSIs. In order to overcome this problem, in this letter, a novel dual-path network (DPN)-based HSI classification method is proposed, in which the DPN combines the advantages of the residual network and dense convolutional network. First, the principal component analysis is utilized to extract significant components of HSI. Second, training image patches centered on labeled pixels are constructed to train the DPN. Finally, the labels of test pixels are predicted by using the trained network. Experiments conducted on two hyperspectral data sets demonstrate the state-of-the-art performance of the proposed method over other compared methods in terms of classification accuracies.

In this article, a novel hyperspectral image (HSI) classification method based on fusing multiple edge-preserving operations (EPOs) is proposed, which consists of the following steps. First, the edge-preserving features are obtained by performing different types of EPOs, i.e., local edge-preserving filtering and global edge-preserving smoothing on the dimension-reduced HSI. Then, with the assistance of a superpixel segmentation method, the edge-preserving features are further improved by considering the inter and intra spectral properties of superpixels. Finally, the spectral and edge-preserving features are fused to form one composite kernel, which is fed into the support vector machine (SVM) followed by a majority voting fusion scheme. Experimental results on three data sets demonstrate the superiority of the proposed method over several state-of-the-art classification approaches, especially when the training sample size is limited. Furthermore, 21 well-known methods, including mathematical morphology-based approaches, sparse representation models, and deep learning-based classifiers, are adopted to be compared with the proposed method on Houston data set with standard sets of training and test samples released during 2013 Data Fusion Contest, which also shows the effectiveness of the proposed method.

The “noisy label” problem is one of the major challenges in hyperspectral image (HSI) classification. In order to address this problem, a spatial density peak (SDP) clustering-based method is proposed to detect mislabeled samples in the training set. Specifically, the proposed methods consist of the following steps: first, the correlation coefficients among the training samples in each class are estimated. In this step, instead of measuring the correlation coefficients by considering individual samples, all neighbor samples or K representative neighbor samples in a local window surrounding each training sample are considered. By this way, the spatial contextual information could be used, and two versions of the proposed method, i.e., measuring the correlation coefficients using all neighbor samples or K representative samples, are referred as SDP and K-SDP, respectively. Second, with the correlation coefficients calculated above, the local density of each training sample can be obtained by the DP clustering algorithm. Finally, those mislabeled samples which usually have lower local densities in each class are able to be identified by a defined decision function. The effectiveness of the proposed detection method is evaluated using a series of spectral and spectral-spatial classification methods on several real hyperspectral data sets.

Mislabeled training samples may have a negative effect on the performance of hyperspectral image classification. In order to solve this problem, a new density peak (DP) clustering-based noisy label detection method is proposed, which consists of the following steps. First, the distances among the training samples of each class are calculated using four representative distance metrics, i.e., the Euclidean distance (ED), orthogonal projection divergence (OPD), spectral information divergence (SID), and correlation coefficient (CC). Then, the local density of each training sample can be obtained using the DP clustering algorithm. Finally, a local density-based decision function is used to detect the noisy labels. The effectiveness of the proposed method is evaluated using the support vector machines on several real hyperspectral data sets. Experimental results demonstrate that the proposed noisy label detection method indeed helps in improving the classification performance.

In this paper, a novel multi-scale structure extraction based spectral-spatial hyperspectral image classification method is proposed, which consists of the following steps. First, the spectral dimension of the hyperspectral image is reduced by averaging adjacent spectral bands. Then, in order to extract the multi-scale significant structural features (MSFs) which are insensitive to image noise and texture, a relative total variation based structure extraction method is applied on the dimension reduced hyperspectral image. Finally, the MSFs are fused together with the kernel principal component analysis (KPCA), so as to obtain the kernel PCA fused multi-scale structural features (KPCA-MSFs) for classification. Experiments conducted on a real hyperspectral image demonstrate the outstanding performance of the proposed approach over several state-of-the-art spectral-spatial classifiers, especially when the image is corrupted by serious scene noise.

In this paper, the effect of ground truths on performance evaluation of hyperspectral image classification is studied. The purpose is to investigate whether the accuracies in terms of three representative accuracy indexes, i.e., the overall accuracy (OA), the average accuracy (AA), and the Kappa coefficient, can be completely responsible when the ground truth is insufficient. The major contribution of this work is designing several experiments so as to subjectively and objectively analysis the influences of ground truths on performance evaluation. Furthermore, four evaluation metrics, i.e., the Pearson linear correlation coefficient (PLCC), root mean square error (RMSE), Spearmans rank correlation coefficient (SR-CC), and Kendalls rank correlation coefficient (KRCC) have been adopted to measure the robustness of different classification methods to ground truths containing different numbers of labeled pixels and the location of ground truth in the image. Based on the designed experiments, a conclusion is obtained that insufficient ground truths may affect the performance of existing accuracy indexes. This underlines that overoptimistic performance evaluations may exist when the ground truth contains a small number of labeled pixels.

In this letter, a novel anomaly detection method is proposed, which can effectively fuse the multiscale information extracted by attribute and edge-preserving filters. The proposed method consists of the following steps. First, multiscale attribute and edge-preserving filters are utilized to obtain multiscale anomaly detection maps. Then, the multiscale detection maps are fused via an averaging approach, and the training samples of the anomalies and background are selected from the fused detection map. Next, the support vector machine classification is performed on the hyperspectral image to obtain an anomaly probability map. Finally, the detection result is obtained by multiplying the fused detection map and the anomaly probability map, followed by an edge-preserving filtering-based postprocessing. Experiments performed on four real hyperspectral data sets demonstrate that the proposed method shows a better detection performance with respect to several state-of-the-art hyperspectral anomaly detection methods.

The joint sparse representation (JSR)-based classifier assumes that pixels in a local window can be jointly and sparsely represented by a dictionary constructed by the training samples. The class label of each pixel can be decided according to the representation residual. However, once the local window of each pixel includes pixels from different classes, the performance of the JSR classifier may be seriously decreased. Since correlation coefficient (CC) is able to measure the spectral similarity among different pixels efficiently, this letter proposes a new classification method via fusing CC and JSR, which attempts to use the within-class similarity between training and test samples while decreasing the between-class interference. First, the CCs among the training and test samples are calculated. Then, the JSR-based classifier is used to obtain the representation residuals of different pixels. Finally, a regularization parameter λ is introduced to achieve the balance between the JSR and the CC. Experimental results obtained on the Indian Pines data set demonstrate the competitive performance of the proposed approach with respect to other widely used classifiers.

Image decolorization is known to be an effective way in transferring a color image into a gray one while well preserving the major information of all three bands. In this paper, a simple yet effective hyperspectral image visualization framework based on decolorization, named decolorization based hyperspectral visualization, is proposed, which enables us to fully exploit the benefits of decolorization technique. The proposed framework consists of the following two main steps. First, the hyperspectral image is partitioned into nine subsets of adjacent hyperspectral bands and the averaged band of each subset is calculated. Then, the dimension reduced image is further divided into three groups of adjacent bands, and the bands in each group are fused by using an image decolorization method. The main contribution of this paper is that the strong correlations in two different fields, i.e., image decolorization and hyperspectral image visualization, are first built. Experiments performed on several real hyperspectral data sets demonstrate that the proposed framework can obtain outstanding visualization performance in terms of both subjective and objective evaluations.

In this paper, a novel method is introduced to detect and correct mislabeled training samples for hyperspectral image classification. First, domain transform recursive filtering-based feature extraction is used to improve the separability of the training samples. Then, constrained energy minimization-based object detection is performed on the training set with each training sample serving as the object spectrum. Finally, the label of each training sample is verified or corrected based on the averaged detection probabilities of different classes. Experiments performed on real hyperspectral data sets demonstrate the effectiveness of the proposed method in improving classification performance with respect to the classifier trained with the original training set that contains a number of mislabeled samples.

The existence of shadows in very high resolution satellite images obstructs image interpretation and the following applications, such as target detection and recognition. Traditional shadow detection methods consider only the pixel-level properties, such as color and intensity of image pixels, and thus, may produce errors around object boundaries. To overcome this problem, a novel shadow detection algorithm based on extended random walker (ERW) is proposed by jointly integrating both shadow property and spatial correlations among adjacent pixels. First, a set of training samples is automatically generated via an improved Otsu-based thresholding method. Then, the support vector machine is applied to obtain an initial detection map, which categorizes all the pixels in the scene into shadow and nonshadow. Finally, the initial detection map is refined with the ERW model, which can simultaneously characterize the shadow property and spatial information in satellite images to further improve shadow detection accuracy. Experiments performed on five real remote sensing images demonstrate the superiority of the proposed method over several state-of-the-art methods in terms of detection accuracy.

In the field of hyperspectral image classification, a widely used way for objective performance evaluation of different classification methods is calculating three accuracy indexes, i.e., the overall accuracy, the average accuracy, and the Kappa coefficient. These accuracy indexes are obtained by comparing the classification results with the ground truth, i.e., a reference classification map labeled by human experts. In this paper, the effect of ground truths on the objective performance evaluation of hyperspectral image classification is studied. The purpose is to investigate, if the ground truth is insufficient, whether the above accuracy indexes can be completely responsible. Furthermore, in order to measure the robustness of different classification methods to those insufficient ground truths, four evaluation metrics, i.e., the Pearson linear correlation coefficient, root-mean-square error, Spearman's rank correlation coefficient, and Kendall's rank correlation coefficient have been adopted for further analysis. Based on these experiments, an interesting conclusion can be obtained that insufficient ground truths may limit the assessment capability of existing accuracy indexes. This underlines that overoptimistic performance evaluations may exist and stresses the demand of designing more appropriate accuracy indexes for objective performance evaluation with insufficient ground truths.

In this paper, a novel spectral-spatial very high resolution images shadow detection algorithm based on random walker is proposed. First, a set of training samples is obtained by an improved Otsu based thresholding method automatically. Then, a widely used pixel-wise classifier, i.e., the Support Vector Machine (SVM), is applied to obtain an initial binary classification map. Finally, the initial classification map is refined with the extended random walker model, which can jointly integrating both the spectral characteristics and spatial-correlation among adjacent pixels to further improve shadow detection accuracy. Experimental results performed on real data sets demonstrate the superiority of the proposed method over several state-of-the-art methods.

Hyperspectral image classification, an astonishing tool to distinguish the land covers in remote sensed hyperspectral images, has been investigated by multiple disciplines such as geoscience, environmental science, mathematics, and computer vision. Following early machine learning (e.g., support vector machines and neural networks) and feature extraction theories (e.g., principal component analysis), hundreds of hyperspectral image classification algorithms have been proposed in order to further improve the classification accuracies. However, it is still unclear what are the real improvements of the newly proposed methods in this field or we are just fitting models to some specific data sets? To address this problem, this paper aims at discussing the major motivations and ideas in conducting a comprehensive benchmark analysis for hyperspectral image classification. The benchmark should not only allows researchers to compare their models with other algorithms but also helps identify the chief factors affecting the performance of their classification methods.

In this paper, a novel model of Gabor Filtering based Deep Network (GFDN) for hyperspectral image classification is proposed. First, spatial features are extracted via Gabor filtering from the three principal components. Gabor filter can capture physical structures of hyperspectral images, such as specific orientation information. Then, the Gabor features and spectral features are simply staked to form combined features. Finally, high-level features are learnt by a stacked sparse auto-encoder deep network. Since the limited training samples negatively affect the classification performance in deep learning, here, an effective way is designed to simulate more training samples. By using both the real and virtual samples, the parameters of deep network can be better learnt and updated, leading to more robust and accurate classification results. Experiments on the real hyperspectral data set reveal the superior performance of the proposed method over some well-known classification methods.

Edge-preserving features (EPFs) obtained by the application of edge-preserving filters to hyperspectral images (HSIs) have been found very effective in characterizing significant spectral and spatial structures of objects in a scene. However, a direct use of the EPFs can be insufficient to provide a complete characterization of spatial information when objects of different scales are present in the considered images. Furthermore, the edge-preserving smoothing operation unavoidably decreases the spectral differences among objects of different classes, which may affect the following classification. To overcome these problems, in this paper, a novel principal component analysis (PCA)-based EPFs (PCA-EPFs) method for HSI classification is proposed, which consists of the following steps. First, the standard EPFs are constructed by applying edge-preserving filters with different parameter settings to the considered image, and the resulting EPFs are stacked together. Next, the spectral dimension of the stacked EPFs is reduced with the PCA, which not only can represent the EPFs in the mean square sense but also highlight the separability of pixels in the EPFs. Finally, the resulting PCA-EPFs are classified by a support vector machine (SVM) classifier. Experiments performed on several real hyperspectral data sets show the effectiveness of the proposed PCA-EPFs, which sharply improves the accuracy of the SVM classifier with respect to the standard edge-preserving filtering-based feature extraction method, and other widely used spectral-spatial classifiers.

A novel method for anomaly detection in hyperspectral images is proposed. The method is based on two ideas. First, compared with the surrounding background, objects with anomalies usually appear with small areas and distinct spectral signatures. Second, for both the background and the objects with anomalies, pixels in the same class are usually highly correlated in the spatial domain. In this paper, the pixels with specific area property and distinct spectral signatures are first detected with attribute filtering and a Boolean map-based fusion approach in order to obtain an initial pixel-wise detection result. Then, the initial detection result is refined with edge-preserving filtering to make full use of the spatial correlations among adjacent pixels. Compared with other widely used anomaly detection methods, the experimental results obtained on real hyperspectral data sets including airport, beach, and urban scenes demonstrate that the performance of the proposed method is quite competitive in terms of computing time and detection accuracy.

Active learning (AL) and semisupervised learning (SSL) are both promising solutions to hyperspectral image classification. Given a few initial labeled samples, this work combines AL and SSL in a novel manner, aiming to obtain more manually labeled and pseudolabeled samples and use them together with the initial labeled samples to improve the classification performance. First, based on a comparison of the segmentation and spectral-spatial classification results obtained by random walker (RW) and extended RW (ERW) algorithms, the unlabeled samples are separated into two different sets, i.e., low- and high-confidence unlabeled data sets. For the high-confidence unlabeled data, pseudolabeling is performed, which can ensure the correctness and informativeness of the pseudolabeled samples. For the low-confidence unlabeled data, AL is used to select samples. In this way, the samples which are more effective for improvement of classification performance can be labeled in only a few iterations. Finally, with the learned training set and the original hyperspectral image as inputs, the ERW classifier is used to obtain the final classification result. Experiments performed on three real hyperspectral data sets show that the proposed method can achieve competitive classification accuracy even with a very limited number of manually labeled samples.

Traditional joint sparse representation based hyperspectral classification methods define a local region for each pixel. Through representing the pixels within the local region simultaneously, the class of the central pixel is able to be decided. A common limitation of this kind of methods is that only local pixels are considered in such methods, and thus, non-local information will be ignored. In order to improve the classification accuracy with the non-local information of hyperspectral images, a novel hyperspectral image classification based on K nearest neighbors (KNN) sparse representation is proposed in this paper. First, a feature space is defined based on the first principal components of the hyperspectral image and the spatial coordinates of different pixels. Then, in the defined feature space, K non-local neighborhoods of each pixel are found by using the KNN searching scheme. At last, through jointly representing the K pixels with the joint sparse model and comparing the representation residuals, the label of each pixel can be determined. Experiments performed on a widely used real HSI data set show that the proposed method obtain better classification performances when compared with the traditional joint sparse representation method and other recently proposed hyperspectral image classification methods.

The 2014 data fusion contest organized by IGARSS 2014 has shown an example that dual images with quite different spatial resolutions may have quite different ground coverages. Specifically, due to the technology limit of optical sensor and the incomplete flight path, the high spatial resolution visible image used in the data fusion contest can only provide a sparse ground coverage. To fill the missing area in this visible image, a simple yet effective image completion method is introduced in this paper which consists of the following two steps: First, through performing patch matching on the hyperspectral image, the most common recurring offsets between patches in the known region and those in the missing region are estimated. Second, the pixels in the missing region is completed by fusing pixels of the shifted visible images (obtained using the above estimated offsets). Experimental results show that the proposed method performs much better than multiple general image completion methods for this data set.

A novel superpixel-based discriminative sparse model (SBDSM) for spectral-spatial classification of hyperspectral images (HSIs) is proposed. Here, a superpixel in a HSI is considered as a small spatial region whose size and shape can be adaptively adjusted for different spatial structures. In the proposed approach, the SBDSM first clusters the HSI into many superpixels using an efficient oversegmentation method. Then, pixels within each superpixel are jointly represented by a set of common atoms from a dictionary via a joint sparse regularization. The recovered sparse coefficients are utilized to determine the class label of the superpixel. In addition, instead of directly using a large number of sampled pixels as dictionary atoms, the SBDSM applies a discriminative K-SVD learning algorithm to simultaneously train a compact representation dictionary, as well as a discriminative classifier. Furthermore, by utilizing the class label information of training pixels and dictionary atoms, a class-labeled orthogonal matching pursuit is proposed to accelerate the K-SVD algorithm while still enforcing high discriminability on sparse coefficients when training the classifier. Experimental results on four real HSI datasets demonstrate the superiority of the proposed SBDSM algorithm over several well-known classification approaches in terms of both classification accuracies and computational speed.

In this paper, a novel feature extraction method based on intrinsic image decomposition (IID) is proposed for hyperspectral image classification. The proposed method consists of the following steps. First, the spectral dimension of the hyperspectral image is reduced with averaging-based image fusion. Then, the dimension reduced image is partitioned into several subsets of adjacent bands. Next, the reflectance and shading components of each subset are estimated with an optimization-based IID technique. Finally, pixel-wise classification is performed only on the reflectance components, which reflect the material-dependent properties of different objects. Experimental results show that, with the proposed feature extraction method, the support vector machine classifier is able to obtain much higher classification accuracy even when the number of training samples is quite small. This demonstrates that IID is indeed an effective way for feature extraction of hyperspectral images.

This paper introduces a novel spectral-spatial classification method for hyperspectral images based on extended random walkers (ERWs), which consists of two main steps. First, a widely used pixelwise classifier, i.e., the support vector machine (SVM), is adopted to obtain classification probability maps for a hyperspectral image, which reflect the probabilities that each hyperspectral pixel belongs to different classes. Then, the obtained pixelwise probability maps are optimized with the ERW algorithm that encodes the spatial information of the hyperspectral image in a weighted graph. Specifically, the class of a test pixel is determined based on three factors, i.e., the pixelwise statistics information learned by a SVM classifier, the spatial correlation among adjacent pixels modeled by the weights of graph edges, and the connectedness between the training and test samples modeled by random walkers. Since the three factors are all well considered in the ERW-based global optimization framework, the proposed method shows very good classification performances for three widely used real hyperspectral data sets even when the number of training samples is relatively small.

We present a novel general-purpose compression method for tomographic images, termed 3D adaptive sparse representation based compression (3D-ASRC). In this paper, we focus on applications of 3D-ASRC for the compression of ophthalmic 3D optical coherence tomography (OCT) images. The 3D-ASRC algorithm exploits correlations among adjacent OCT images to improve compression performance, yet is sensitive to preserving their differences. Due to the inherent denoising mechanism of the sparsity based 3D-ASRC, the quality of the compressed images are often better than the raw images they are based on. Experiments on clinical-grade retinal OCT images demonstrate the superiority of the proposed 3D-ASRC over other well-known compression methods.

From a single low resolution image, a real-time document image super-resolution algorithm is proposed to obtain high resolution document image with sharp text boundaries. First, a highly efficient document image matting algorithm based on local linear modeling is designed to decompose the input image into text, foreground and background layers, which contain the text edge information, the color information of the foreground and background respectively. Then the text layer is up-sampled with Teager filter to increase the sharpness of the text. For efficiency, the foreground and background layers are simply up-sampled through the bi-cubic interpolation. Finally, these three high resolution layers are composed to obtain the high-resolution image. Experiments on real scanned document images demonstrate the effectiveness of the proposed method in both visual perception and OCR performance

This paper proposes a new spectral-spatial hyperspectral classification method named the shape-adaptive sparse representation (SASR). The fixed window is not suitable for all pixels of hyperspectral image (HSI) to search local similar regions. In order to overcome the drawback, we propose to apply the shape-adaptive algorithm to exploit the contextual spatial information of HSI. Furthermore, the hyperspectral classification is implemented by incorporating the spatial contextual information of HSI into the sparse representation classification model. Experimental results demonstrate the superiority of the proposed SASR method over both classical and state-of-the-art approaches.

A novel spectral-spatial hyperspectral image classification is proposed based on extended random walkers. First, a widely used pixel-wise classifier, i.e., the support vector machine (SVM), is adopted to obtain probability maps for a hyper-psectral image, which measure the probabilities that a pixel belongs to different classes. Then, the initial probabilities are optimized with the extended random walkers. Finally, by assigning each pixel with the label for which the greatest probability is obtained, the classification result is obtained. Experiments show the outstanding performance of the proposed method in terms of classification accuracy especially when the number of training samples is relatively small.

Sparse representation has been demonstrated to be a powerful tool in classification of hyperspectral images (HSIs). The spatial context of an HSI can be exploited by first defining a local region for each test pixel and then jointly representing pixels within each region by a set of common training atoms (samples). However, the selection of the optimal region scale (size) for different HSIs with different types of structures is a nontrivial task. In this paper, considering that regions of different scales incorporate the complementary yet correlated information for classification, a multiscale adaptive sparse representation (MASR) model is proposed. The MASR effectively exploits spatial information at multiple scales via an adaptive sparse strategy. The adaptive sparse strategy not only restricts pixels from different scales to be represented by training atoms from a particular class but also allows the selected atoms for these pixels to be varied, thus providing an improved representation. Experiments on several real HSI data sets demonstrate the qualitative and quantitative superiority of the proposed MASR algorithm when compared to several well-known classifiers.

The integration of spatial context in the classification of hyperspectral images is known to be an effective way in improving classification accuracy. In this paper, a novel spectral-spatial classification framework based on edge-preserving filtering is proposed. The proposed framework consists of the following three steps. First, the hyperspectral image is classified using a pixelwise classifier, e.g., the support vector machine classifier. Then, the resulting classification map is represented as multiple probability maps, and edge-preserving filtering is conducted on each probability map, with the first principal component or the first three principal components of the hyperspectral image serving as the gray or color guidance image. Finally, according to the filtered probability maps, the class of each pixel is selected based on the maximum probability. Experimental results demonstrate that the proposed edge-preserving filtering based classification method can improve the classification accuracy significantly in a very short time. Thus, it can be easily applied in real applications.

Feature extraction is known to be an effective way in both reducing computational complexity and increasing accuracy of hyperspectral image classification. In this paper, a simple yet quite powerful feature extraction method based on image fusion and recursive filtering (IFRF) is proposed. First, the hyperspectral image is partitioned into multiple subsets of adjacent hyperspectral bands. Then, the bands in each subset are fused together by averaging, which is one of the simplest image fusion methods. Finally, the fused bands are processed with transform domain recursive filtering to get the resulting features for classification. Experiments are performed on different hyperspectral images, with the support vector machines (SVMs) serving as the classifier. By using the proposed method, the accuracy of the SVM classifier can be improved significantly. Furthermore, compared with other hyperspectral classification methods, the proposed IFRF method shows outstanding performance in terms of classification accuracy and computational efficiency.

Pansharpening aims at creating a fused image of high spatial and spectral resolutions through merging a panchromatic (PAN) image with a multispectral (MS) image. Component substitution is the most widely used pansharpening method. However, most research in this field focuses on improving the existing component substitution-based pansharpening methods, e.g., principal component substitution and intensity hue saturation transform. The major contribution of this paper is a novel component substitution framework based on an image matting model. The matting model refers to an MS image that can be decomposed into three components, i.e., alpha channel, spectral foreground, and background. Through substituting the alpha channel of the MS image with the PAN image, the high-resolution MS image is able to be reconstructed perfectly. Experiments performed on different data sets demonstrate that the proposed method outperforms several state-of-the-art pansharpening methods in terms of subjective and objective evaluation.

A novel matting model based pansharpening (MMP) method is proposed for creating a fused image of high spatial and spectral resolutions through merging a panchromatic (PAN) image with a multi-spectral (MS) image. Matting model refers to that an MS image can be decomposed into three components, i.e., alpha channel, spectral foreground and background. Through substituting the alpha channel of the MS image with the PAN image, the edge information from the PAN image can be transferred into the MS image without causing spectral distortion. Experiments demonstrate the superiority of the proposed method by subjective and objective evaluation.

A fast and effective image fusion method is proposed for creating a highly informative fused image through merging multiple images. The proposed method is based on a two-scale decomposition of an image into a base layer containing large scale variations in intensity, and a detail layer capturing small scale details. A novel guided filtering-based weighted average technique is proposed to make full use of spatial consistency for fusion of the base and detail layers. Experimental results demonstrate that the proposed method can obtain state-of-the-art performance for fusion of multispectral, multifocus, multimodal, and multiexposure images.