Although remote sensing (RS) data with multiple modalities can be used to significantly improve the accuracy of semantic segmentation in RS data, how to effectively extract multimodal information through multimodal feature fusion remains a challenging task. Specifically, existing methods for multimodal feature fusion still face two major challenges: 1) Due to the diverse imaging mechanisms of multimodal RS data, the boundaries of the same foreground may vary across different modalities, leading to the inclusion of unwanted background semantics in the fused foreground features; 2) RS data from different modalities exhibit varying discriminative abilities for different foregrounds, making it challenging to determine the proportion of semantic information for each modality in the fusion results. To address the above issues, we propose a dynamic feature fusion method based on region-wise queries, namely DF2RQ, for SS of multimodal RS data. This method is primarily composed of two components: the spatial reconstruction (SR) module and the dynamic fusion (DF) module. Within the SR module, we propose a spatial reconstruction scheme that samples foreground features from different modalities, achieving independent reconstruction of different unimodal features, thereby alleviating the semantic mixing between foreground and background across modalities. In the DF module, a feature fusion scheme based on unimodal feature reference positions is proposed to obtain fusion weights for each modality, thereby enabling the dynamic fusion of complementary features from multiple modalities. The performance of the proposed method has been extensively evaluated on various multimodal RS datasets for SS, and the experimental results consistently show that the proposed method achieves state-of-the-art accuracy on multiple commonly used metrics. In addition, our code is available at https://github.com/I3ab/DF2RQ.

Recent advancements in hyperspectral anomaly detection (HAD) utilizing deep learning have garnered significant attention due to their superior performance. However, most existing methods based on convolutional neural networks and Transformer focus on extracting local and global features separately and assume that the background resides in a single subspace for reconstruction, thereby reducing the quality of the reconstructed background and decreasing the accuracy of HAD. Moreover, although incorporating prior physical knowledge into the loss function can enhance the performance of the deep learning networks, it also increases the number of hyperparameters and complicates the tuning process. To address these issues, we propose a deep prior Mamba network (DPMN) for HAD, which primarily consists of two components: the abundance generation module (AGM) and the background reconstruction module (BRM). Specifically, AGM employs convolution layers to extract local information and introduces Mamba to capture long-range dependencies, achieving feature extraction from local to global. Subsequently, BRM utilizes a learnable background dictionary to divide the background into multiple subspaces for reconstruction, realizing accurate background reconstruction while effectively suppressing the interference of anomalies on background reconstruction. Furthermore, to fully leverage the intrinsic properties of hyperspectral images, we incorporate a regularization term into the loss function, merging the total variation (TV) with the low-rank representation (LRR), which not only exploits spatial smoothness and low-rankness but also reduces the number of hyperparameters. Experimental results on eight publicly available real datasets demonstrate that our method significantly outperforms other state-of-the-art methods. In addition, our code is available at: https://github.com/I3ab/DPMN.

This paper proposes an autoencoder (AE) framework with transformer encoder and extended multilinear mixing model (EMLM) embedded decoder for nonlinear hyperspectral anomaly detection. Specifically, the proposed AE frame-work adopts the transformer as the encoder so that not only the local spatial information, but also the transitive global spatial information can be considered, and the EMLM is embedded into the decoder to accurately characterize the high-order nonlinear mixing phenomenon. By using this AE framework, the background of HSIs can be reconstructed accurately. Finally, the anomalous level of pixel is computed by the reconstruction error. The experimental results on two real hyperspectral datasets demonstrate that the proposed method outperforms the current state-of-the-art (SOTA) anomaly detectors.

Recent advancements in weakly supervised point cloud semantic segmentation have diminished the reliance on extensive annotations, thereby enhancing the efficacy of understanding the real-world environment. However, existing approaches, such as PSD [25], SQN [5] and OTOC [10], often overlook the valuable global class-related prior knowledge present in point clouds beyond the scope of labels. To fully leverage this prior knowledge, which suggests that points of the same class should be close in feature space and each class should have a representative feature, we propose G-Former, a grouping transformer model. G-Former incorporates the idea of clustering into the overall model by defining clusters aligned to classes and assigning learnable tensors as cluster centers. Points are then grouped into these clusters based on the similarity of their features to the cluster centers. The core components of G-Former include a Hierarchy Cluster Structure (HCS) and a Grouping Module (GM). The former consists of two sets of clusters, one for classes while the other serves as a middle layer to help class clusters handle large-scale point features. The latter facilitates grouping the point cloud into different clusters. With the help of the grouping transformer model, G-Former further proposes a series of cluster center constraints to augment inter-class distances and diminish intra-class distances to enhance the discriminability of points. Experimental results on ScanNet v2 and S3DIS datasets demonstrate that G-Former outperforms previous methods with limited labels (0.1% or 1%) by a significant margin and is even comparable to fully supervised methods.

Object detection in remote sensing images (RSIs) remains a challenging task due to complex variations in object scale, dense arrangements, and arbitrary orientations. Compared to the widely used multistage and one-stage approaches, query-based methods that avoid postprocessing procedures and implement end-to-end inference, have recently attracted much attention. However, existing query-based methods still face two main challenges: 1) the feature sampling regions predicted by the query vectors often fail to be aligned with the foreground features, making it difficult to accurately classify and locate potential objects; and 2) the cascade decoders are crucial for optimizing the query vectors, resulting in a slower inference process. To address the above issues, we propose a novel object detection method named point set decoding based on semantic query (PSD-SQ), which mainly consists of two components: a semantic query generator (SQG) module and an oriented point set decoder (OPSD) module. The SQG module is proposed to generate semantic query vectors with rich object information based on the semantic correlations among feature vectors. The OPSD module includes two blocks: a point sampling with angle (PSA) block and a dynamic interactor (DI) block. The PSA block is constructed to refine the sampling locations with predicted angles, aligning the sampling locations and oriented object regions, and the DI block is designed to decode the sampled features with dynamic weights, making the decoding process more efficient. The proposed method is extensively evaluated on various object detection datasets of RSIs, and the experimental results consistently demonstrate that the proposed method achieves state-of-the-art (SOTA) performance in terms of both accuracy and inference speed. In addition, our code is available athttps://github.com/I3ab/PSD_SQ.

Although existing unsupervised domain adaptation (UDA) methods have successfully applied to semantic segmentation tasks for high-resolution remote sensing (HRS) images, they still have some limitations that need to be addressed: 1) they mainly focus on aligning the marginal distributions while ignoring the interdomain differences in the conditional distributions, which may be suboptimal because they assume that the boundaries of category decision are identical across domains; and 2) they depend on self-supervised learning for easy-to-hard alignment, which may result in model learning erroneous knowledge from the pseudo labels. To address the above limitations, we propose a joint distribution adaptive-alignment framework (JDAF) to eliminate the distribution difference between the source and target domains, which is mainly composed of a marginal distribution alignment (MDA) module, a conditional distribution alignment (CDA) module, and an improved easy-to-hard adaptation strategy. The MDA module is used to narrow local semantic and global spatial differences between domains and first advance, and then, the CDA module that includes a category-invariant feature alignment (CFA) block and a dataset-level context aggregation (DCA) block is presented and designed, which can dynamically update and align the feature representations that are invariant to category change and adaptively incorporate dataset-level context into the features of source domain to enhance the pixel-level representation. An uncertainty-adaptive learning (UAL) method is, moreover, proposed to improve the easy-to-hard adaptation strategy by enabling the model to learn accurate knowledge from the pseudo labels, which can boost the adaptive performance of the whole JDAF. Comprehensive experiments with four cross-domain tasks on two benchmark datasets of aerospace HRS images demonstrate that the proposed JDAF achieves significant performance gains compared to the state-of-the-art cross-domain semantic segmentation methods. Our code is available at:https://github.com/maple-hx/JDAF.

To mitigate the impact of mixed pixels in hyperspectral images (HSIs), substantial progress has been made in both model- and deep learning-based unmixing methods. However, issues such as complex computational processes and limited interpretability, hinder the improvement of their unmixing performance. Particularly, unsupervised nonlinear hyperspectral unmixing (HU) remains a great challenge. In this paper, we propose an extended multilinear mixing (EMLM) model-inspired dual-stream network for unsupervised nonlinear HU. Firstly, the alternating direction method of multipliers (ADMM) algorithm for the EMLM-based unmixing problem is unfolded to construct an encoder network. Subsequently, it is connected to a decoder network derived from the EMLM, creating an autoencoder-like network architecture. Secondly, the original HSIs and superpixel-averaging-based coarse HSIs are input into two network branches with identical architectures, respectively, to build a novel weight-sharing dual-stream network. Furthermore, estimates of abundances and nonlinear parameters obtained from the two branches are utilized to formulate local spatial similarity regularizers, enhancing the network’s loss function and effectively improving unmixing accuracy. Finally, experiments conducted on the laboratory-created dataset and real-world datasets validate that the proposed method exhibits superior unmixing performance compared to state-of-the-art methods. In addition, our code is available at: https://github.com/I3ab/EMLM-Net.

Recently, the autoencoder (AE) has received significant attention in the hyperspectral anomaly detection task. However, all existing AE-based anomaly detectors operate under the linear mixing model, which cannot accurately model the nonlinear mixing phenomenon in practical hyperspectral images (HSIs). Moreover, these AE-based detectors rarely consider the spatial information between pixels, which is crucial to obtain accurate results of anomaly detection. To address the above issues, this paper proposes a transformer-based AE framework (TAEF) for nonlinear hyperspectral anomaly detection. Specifically, the proposed AE framework adopts the transformer as the encoder so that not only the local spatial information, but also the transitive global spatial information can be considered. And the extended multilinear mixing model (EMLM) is embedded into the decoder to accurately characterize the high-order nonlinear mixing phenomenon. By using this transformer-based AE framework, the background of HSIs can be reconstructed effectively. Moreover, a novel method for generating patches is proposed in this paper to support the transformer in the characterization of the transitive global spatial information. Besides, to further improve the accuracy of the background reconstruction, the local-clustering method is adopted to decrease the potential anomalies and increase the sparse backgrounds in the meantime. Finally, the anomalous level of pixel is calculated by the reconstruction error. The experimental results on various real hyperspectral datasets demonstrate that the proposed TAEF outperforms the current state-of-the-art anomaly detectors. In addition, our code is available at: https://github.com/I3ab/TAEF.

The effectiveness of spectral-spatial feature learning is crucial for the hyperspectral image (HSI) classification task. Diffusion models, as a new class of groundbreaking generative models, have the ability to learn both contextual semantics and textual details from the distinct timestep dimension, enabling the modeling of complex spectral-spatial relations in HSIs. However, existing diffusion-based HSI classification methods only utilize manually selected single-timestep single-stage features, limiting the full exploration and exploitation of rich contextual semantics and textual information hidden in the diffusion model. To address this issue, we propose a novel diffusion-based feature learning framework that explores Multi-Timestep Multi-Stage Diffusion features for HSI classification for the first time, called MTMSD. Specifically, the diffusion model is first pretrained with unlabeled HSI patches to mine the connotation of unlabeled data, and then is used to extract the multi-timestep multi-stage diffusion features. To effectively and efficiently leverage multi-timestep multi-stage features, two strategies are further developed. One strategy is class & timestep-oriented multi-stage feature purification module with the inter-class and inter-timestep prior for reducing the redundancy of multi-stage features and alleviating memory constraints. The other one is selective timestep feature fusion module with the guidance of global features to adaptively select different timestep features for integrating texture and semantics. Both strategies facilitate the generality and adaptability of the MTMSD framework for diverse patterns of different HSI data. Extensive experiments are conducted on four public HSI datasets, and the results demonstrate that our method outperforms state-of-the-art methods for HSI classification, especially on the challenging Houston 2018 dataset. The codes are available at https://github.com/zjyaccount/MTMSD.

Deep learning (DL)-based methods have been widely used in pansharpening and have made great progress. To increase the accuracy, the DL-based model structures can be improved by introducing the multiresolution information and self-similarity of the panchromatic (PAN) image and multispectral (MS) images, respectively, but few methods exist to fully exploit both the characteristics in the constructed models. To solve the above problem, this letter proposes a PAN-guided multiresolution fusion (PMRF) network based on Swin transformer (ST). In the proposed PMRF network, the multiresolution features extracted from the PAN image are fused with the features extracted from the MS images to guide the level-by-level improvement in the spatial resolution. Furthermore, a ST-based residual self-attention (STRA) module is designed to combine the advantages of ST and residual learning to fully exploit the self-similarity to enhance the feature representation. Experimental results show that the proposed method outperforms the state-of-the-art methods in both spatial enhancement and spectral preservation.

Global channel pruning (GCP) aims to remove a subset of channels (filters) across different layers from a deep model without hurting the performance. Previous works focus on either single task model pruning or simply adapting it to multitask scenario, and still face the following problems when handling multitask pruning: 1) Due to the task mismatch, a well-pruned backbone for classification task focuses on preserving filters that can extract category-sensitive information, causing filters that may be useful for other tasks to be pruned during the backbone pruning stage; 2) For multitask predictions, different filters within or between layers are more closely related and interacted than that for single task prediction, making multitask pruning more difficult. Therefore, aiming at multitask model compression, we propose a Performance-Aware Global Channel Pruning (PAGCP) framework. We first theoretically present the objective for achieving superior GCP, by considering the joint saliency of filters from intra- and inter-layers. Then a sequentially greedy pruning strategy is proposed to optimize the objective, where a performance-aware oracle criterion is developed to evaluate sensitivity of filters to each task and preserve the globally most task-related filters. Experiments on several multitask datasets show that the proposed PAGCP can reduce the FLOPs and parameters by over 60% with minor performance drop, and achieves 1.2x ∼ 3.3x acceleration on both cloud and mobile platforms. Our code is available at http://www.github.com/HankYe/PAGCP.git.

Recently, the research on nonlinear unmixing for hyperspectral images (HSIs) has received more and more attention. However, unsupervised nonlinear unmixing methods that jointly estimate endmembers and abundances from HSIs are insufficiently studied. Besides, the reasonable description of the wavelength-dependent nonlinear intensity and the effective utilization of the spectral and spatial information of HSIs remain to be improved. Based on an extended multilinear mixing model, a coarse-to-fine scheme is proposed for unsupervised nonlinear hyperspectral unmixing to address the above issues. Coarse HSIs generated based on the superpixel segmentation are unmixed first, and then fine unmixing on the original HSIs is achieved with the guidance of the coarse unmixing results. The endmembers extracted by the coarse unmixing are used to update the endmembers in the fine unmixing, and the coarse abundances and nonlinear parameters are integrated into regularizers. To be specific, a weighted sparse regularizer of abundances and a weighted graph regularizer of nonlinear parameters are constructed and incorporated into the objective function. In this way, some priors can be well modeled and exploited, including that the neighboring pixels share similar sparsity patterns in the abundances and show the consistency in correlations between different bands of the nonlinear parameters. Finally, the alternative optimization strategy and the alternating direction method of multipliers are applied to derive the algorithm. Experimental results on the synthetic, laboratory-created, and real hyperspectral data demonstrate that the proposed method outperforms the state-of-the-art nonlinear unmixing methods.

Due to noisy acquisition and atmospheric effects, some spectral bands in hyperspectral images (HSIs) suffer from low signal-to-noise ratios, thus requiring robust techniques to tackle unmixing problems. Besides, integrating the spatial information of HSIs into the nonlinear unmixing framework remains a challenge. To cope with the above problems, first, the$\ell_{2.1}$norm-based objective function is adopted to suppress the influence of noisy bands. Furthermore, to fully exploit the spatial-spectral information of HSIs, a reweighted collaborative sparse regularizer imposed on the abundances enforces that the pixels in a superpixel-based neighborhood share the same set of endmembers and have similar abundances, and a reweighted spectral total variation regularizer is employed to enhance the spatial-spectral smoothness of the nonlinear parameters. Extensive experiments conducted on the simulated and real datasets verify the superiority of the proposed algorithm over other state-of-the-art ones.

This paper presents a novel kernel-based decomposition model with total variation and sparsity regularizations via union dictionary for nonlinear hyperspectral anomaly detection. It decomposes a hyperspectral imagery into three components: background, anomaly, and noise. By using a union dictionary consisting of background and potential anomalous pixels, each test pixel can be well represented. Further, by utilizing endmember-kernel theory to handle nonlinear interactions between atoms in the dictionary, the complex light scattering effects can be effectively characterized. Besides, to separate these components effectively, the total variation and sparsity regularizations are incorporated into the decomposition model to represent the spatial properties of the background and the anomaly, respectively. The experimental results on simulated and real hyperspectral data sets demonstrated the effectiveness of our proposed method compared to several conventional and state-of-the-art anomaly detection methods.

Improving the performance of nonlinear unmixing has become an active topic among the remote sensing applications. Usually, the noise levels of hyperspectral images (HSIs) vary with different bands. However, this fact is generally ignored and may, to some extent, result in a degradation of the unmixing results. Nonetheless, valuable spatial information that provides a great potential for improving the performance has seldom been considered in the current nonlinear unmixing. In this letter, we propose a novel kernel-based nonlinear unmixing model in which the band-wise noise characterization and the spatial relationships of HSIs are incorporated to solve the above problems. Firstly, the noise levels of different bands are estimated based on the results of superpixel segmentation, and then they are used to characterize the roles of different bands in the unmixing process. To exploit the spatial relationships in the superpixels, a regional${\ell }_{1} {-\text {norm}}$regularization is proposed and incorporated into the unmixing model. Experimental results on both synthetic and real hyperspectral datasets demonstrate the superiority of the proposed model compared to the state-of-the-art nonlinear unmixing methods.

Unsupervised domain adaptive object detection aims to adapt a well-trained detector from its original source domain with rich labeled data to a new target domain with unlabeled data. Previous works focus on improving the domain adaptability of region-based detectors,e.g., Faster-RCNN, through matching cross-domain instance-level features that are explicitly extracted from a region proposal network (RPN). However, this is unsuitable for region-free detectors such as single shot detector (SSD), which perform a dense prediction from all possible locations in an image and do not have the RPN to encode such instance-level features. As a result, they fail to align important image regions and crucial instance-level features between the domains of region-free detectors. In this work, we propose an adversarial module, namely, densely semantic enhancement module (DSEM), to strengthen the cross-domain matching of instance-level features for region-free detectors. Firstly, to emphasize the important regions of image, the DSEM learns to predict a transferable foreground enhancement mask that can be utilized to suppress the background disturbance in an image. Secondly, considering that region-free detectors recognize objects of different scales using multi-layer feature maps, the DSEM encodes multi-scale representations across different domains. Finally, the DSEM is pluggable into different region-free detectors, ultimately achieving the densely semantic feature matching via adversarial learning. Extensive experiments have been conducted on PASCAL VOC, Clipart, Comic, W atercolor, and FoggyCityscape benchmarks, and their results well demonstrate that the proposed approach not only improves the domain adaptability of region-free detectors but also outperforms existing domain adaptive region-based detectors under various domain shift settings.

Anomaly detection has been known to be an important issue in hyperspectral remote sensing applications. It aims to detect anomalous targets whose spectral signatures are very different from the background pixels. Although many linear detectors have obtained acceptable detection results, the linear model might not be able to describe complex hyperspectral data and could be replaced by nonlinear models. In this article, we investigate the intrinsic nonlinear characteristics of hyperspectral images (HSIs) on basis of the nonlinear mixing models and propose a novel nonlinear hyperspectral anomaly detection method based on kernel theory and union dictionary. First, the global strong anomalies in the scene and the local background pixels are utilized to construct a union dictionary. Then, a nonlinear representation-based anomaly detection model with the constructed union dictionary is designed, in which the nonlinear mixing effect of HSIs is considered. Meanwhile, the kernel theory is exploited to deal with the nonlinear interactions among the atoms in the dictionary. Finally, the anomalous level of a test pixel is determined by the representation coefficients associated with the anomaly dictionary. The proposed method is evaluated on both synthetic and real hyperspectral datasets. Experimental results demonstrate its excellent performance in comparison with linear and nonlinear state-of-the-art anomaly detectors.

Due to the presence of multiple scatterings, linear unmixing methods may not perform well in practical applications, and thus nonlinear unmixing has become an urgent problem to be solved. Usually, the mixing process in the observed scenarios is physically based, and many well-designed models have been proposed to interpret it. Recently, kernel-based nonlinear unmixing methods have been popularly studied to achieve a model-free and flexible representation of the nonlinearity. However, the existing kernel-based methods are mainly data-driven, which could make them fail to match the real physical mixing mechanism and result in the occurrence of overfitting. In this article, a kernel-based bilinear unmixing (KBU) method was proposed to transform the classic bilinear mixing models into their equivalent kernel forms that are more general and effective in expressing second-order scatterings. Two specific types of kernel transformations were designed, and the alternating direction method of multipliers (ADMM) was used to solve the kernel-transformed model-based optimization problem for unmixing. Moreover, the spatial prior was exploited to further improve the unmixing accuracy, and here we employ the total variation (TV) regularization as a paradigm. Experiments on synthetic data sets, physics-based simulated data sets, and real data were conducted to evaluate the algorithms. It is validated that our methods have better performance in abundance estimation and nonlinear reconstruction compared with other nonlinear unmixing methods.

Recently, nonlinear unmixing algorithms have attracted special attention in hyperspectral image (HSI) processing. However, the inherent wavelength-dependent nonlinear intensity and noise effects in real HSIs are often overlooked, and the spatial information of HSIs has not been fully utilized in current studies. In this paper, we propose a spectral-spatial reweighted robust nonlinear unmixing algorithm to solve the above problems. First, a robust unmixing method is built on an extended multilinear mixing model (EMLM), which employs the vectorized nonlinear parameters to describe the nonlinear intensity varying along with spectral bands, and adopts thel2,1norm-based loss function to suppress the influence of noise. Second, to fully exploit the spectral-spatial information of HSIs, the nonlinear unmixing problem is reformulated with two regularizers. Specifically, a reweighted collaborative sparse regularizer is used to make the pixels in a superpixel-based neighborhood share the same subset of endmembers and have similar abundances because the neighboring pixels are usually composed of several materials in similar proportions, and a reweighted spectral total variation regularizer is utilized to improve the spectral-spatial smoothness of the vectorized nonlinear parameters by considering the local-region similarities of the nonlinear mixing effects. Finally, the constrained optimization problem is solved by the alternating direction method of multipliers (ADMM). Experimental results on simulated, semi-simulated, and real hyperspectral datasets demonstrate that the proposed method outperforms several state-of-the-art nonlinear unmixing methods.

Many linear approaches have been extensively proposed for the anomaly detection problem in hyperspectral images (HSIs), while nonlinear approaches have been rarely studied although most practical cases are nonlinear. Moreover, these existing nonlinear methods simply nonlinearly map each pixel into a high-dimensional space, which does not describe complex light scattering effects between endmembers. To address the above issues, this paper proposes an endmember-kernel-based decomposition model with total variation and sparsity regularizations via union dictionary for the nonlinear anomaly detection in HSIs. The proposed decomposition model utilizes endmember-kernel theory to handle nonlinear interactions between atoms in the dictionary, allowing for effective characterization of complex light scattering effects. By using this endmember-kernel-based decomposition model, a hyperspectral imagery can be decomposed into three components: anomaly, background, and noise. To separate these components effectively, the total variation (TV) and sparsity regularizations are incorporated into the decomposition model to characterize the spatial properties of the background and the anomaly, respectively. Besides, we present a novel construction framework of union dictionary that combines superpixel segmentation and clustering methods sequentially to achieve more accurate dictionary representation capabilities. Finally, the anomalous level of a tested pixel is calculated by the abundances associated with the anomaly dictionary. The experimental results on both synthetic and real hyperspectral data sets demonstrate that the proposed method outperforms several linear and nonlinear state-of-the-art anomaly detectors.

Although domain adaptation has been extensively studied in natural image-based segmentation tasks, the research on cross-domain segmentation for very-high-resolution (VHR) remote sensing images (RSIs) still remains underexplored. The VHR RSI-based cross-domain segmentation mainly faces two critical challenges: 1) large area land covers with many diverse object categories bring severe local patch-level data distribution deviations, thus yielding different adaptation difficulties for different local patches and 2) different VHR sensor types or dynamically changing modes cause the VHR images to go through intensive data distribution differences even for the same geographical location, resulting in different global feature-level domain gaps. To address these challenges, we propose a curriculum-style local-to-global cross-domain adaptation framework for the segmentation of VHR RSIs. The proposed curriculum-style adaptation performs the adaptation process in an easy-to-hard way according to the adaptation difficulties that can be obtained using an entropy-based score for each patch of the target domain and, thus, well aligns the local patches in a domain image. The proposed local-to-global adaptation performs the feature alignment process from the locally semantic to globally structural feature discrepancies and consists of a semantic-level domain classifier and an entropy-level domain classifier that can reduce the above cross-domain feature discrepancies. Extensive experiments have been conducted in various cross-domain scenarios, including geographic location variations and imaging mode variations, and the experimental results demonstrate that the proposed method can significantly boost the domain adaptability of segmentation networks for VHR RSIs.

Semi-supervised few-shot learning aims to improve the model generalization ability by means of both limited labeled data and widely-available unlabeled data. Previous works attempt to model the relations between the few-shot labeled data and extra unlabeled data, by performing a label propagation or pseudo-labeling process using an episodic training strategy. However, the feature distribution represented by the pseudo-labeled data itself is coarse-grained, meaning that there might be a large distribution gap between the pseudo-labeled data and the real query data. To this end, we propose a sample-centric feature generation (SFG) approach for semi-supervised few-shot image classification. Specifically, the few-shot labeled samples from different classes are initially trained to predict pseudo-labels for the potential unlabeled samples. Next, a semi-supervised meta-generator is utilized to produce derivative features centering around each pseudo-labeled sample, enriching the intra-class feature diversity. Meanwhile, the sample-centric generation constrains the generated features to be compact and close to the pseudo-labeled sample, ensuring the inter-class feature discriminability. Further, a reliability assessment (RA) metric is developed to weaken the influence of generated outliers on model learning. Extensive experiments validate the effectiveness of the proposed feature generation approach on challenging one- and few-shot image classification benchmarks.

Unsupervised domain adaptive object detection aims to adapt a well-trained detector from its original source domain with rich labeled data to a new target domain with unlabeled data. Recently, mainstream approaches perform this task through adversarial learning, yet still suffer from two limitations. First, they mainly align marginal distribution by unsupervised cross-domain feature matching, and ignore each feature's categorical and positional information that can be exploited for conditional alignment; Second, they treat all classes as equally important for transferring cross-domain knowledge and ignore that different classes usually have different transferability. In this article, we propose a joint adaptive detection framework (JADF) to address the above challenges. First, an end-to-end joint adversarial adaptation framework for object detection is proposed, which aligns both marginal and conditional distributions between domains without introducing any extra hyper-parameter. Next, to consider the transferability of each object class, a metric for class-wise transferability assessment is proposed, which is incorporated into the JADF objective for domain adaptation. Further, an extended study from unsupervised domain adaptation (UDA) to unsupervised few-shot domain adaptation (UFDA) is conducted, where only a few unlabeled training images are available in unlabeled target domain. Extensive experiments validate that JADF is effective in both the UDA and UFDA settings, achieving significant performance gains over existing state-of-the-art cross-domain detection methods.

The goal of few-shot fine-grained image classification is to recognize rarely seen fine-grained objects in the query set, given only a few samples of this class in the support set. Previous works focus on learning discriminative image features from a limited number of training samples for distinguishing various fine-grained classes, but ignore one important fact that spatial alignment of the discriminative semantic features between the query image with arbitrary changes and the support image, is also critical for computing the semantic similarity between each support-query pair. In this work, we propose an object-aware long-short-range spatial alignment approach, which is composed of a foreground object feature enhancement (FOE) module, a long-range semantic correspondence (LSC) module and a short-range spatial manipulation (SSM) module. The FOE is developed to weaken background disturbance and encourage higher foreground object response. To address the problem of long-range object feature misalignment between support-query image pairs, the LSC is proposed to learn the transferable long-range semantic correspondence by a designed feature similarity metric. Further, the SSM module is developed to refine the transformed support feature after the long-range step to align short-range misaligned features (or local details) with the query features. Extensive experiments have been conducted on four benchmark datasets, and the results show superior performance over most state-of-the-art methods under both 1-shot and 5-shot classification scenarios.

Anomaly detection is an important task among hyperspectral applications, which detects the anomalous targets that are spectrally different from their surroundings. Most of the anomaly detectors construct the detection models in the linear space, where the nonlinear characteristics of data are not taken into consideration. The kernel-based methods, which implicitly map the data into a high-dimensional feature space, have shown great potential in dealing with nonlinear problems. Nevertheless, the original kernel-based detection methods only exploit the spectral features without taking advantage of the spatial correlations among adjacent pixels. In this letter, a nonlinear spectral-spatial composite kernel-based detector (SSCKD) is proposed for hyperspectral anomaly detection. First, superpixel segmentation is adopted to extract spatial features according to local homogeneity. Then, considering the nonlinear characteristics in the hyperspectral data, spectral and spatial features are combined by the composite kernel. Finally, an iterative composite kernel-learning procedure based on the centered kernel alignment is designed to adaptively determine the kernel weights. The experiments are conducted on three real hyperspectral data sets. The detection results demonstrate the superiority of the proposed method in comparison with the conventional and state-of-the-art anomaly detection methods.

For hyperspectral images (HSIs), the imbalance between the high dimensionality and the limited labeled samples has been a main obstacle to classification task. As a solution, semisupervised learning utilizing both labeled and unlabeled samples has shown its potential. In this letter, a novel semisupervised classification framework based on graph attention networks (GATs) for HSIs is proposed. Spatial-spectral joint measurement is designed for the graph model construction to make full use of spatial information. In the convolution process, different weights are assigned to different neighboring nodes according to their attention coefficients, avoiding designing connection weights artificially in previous graph convolution networks (GCNs). Experimental results on multiple hyperspectral data sets with various contexts and resolutions demonstrate that the proposed method outperforms several state-of-the-art graph-based methods.

Anomaly detection in hyperspectral imagery has been an active topic among the remote sensing applications. It aims at identifying anomalous targets with different spectra from their surrounding background. Therefore, an effective detector should be able to distinguish the anomalies, especially for the weak ones, from the background and noise. In this article, we propose a novel method for hyperspectral anomaly detection based on total variation (TV) and sparsity regularized decomposition model. This model decomposes the hyperspectral imagery into three components: background, anomaly, and noise. In order to distinguish effectively these components, a union dictionary consisting of both background and potential anomalous atoms is utilized to represent the background and anomalies, respectively. Moreover, the TV and the sparsity-inducing regularizations are incorporated to facilitate the separation. Besides, we present a new strategy for constructing the union dictionary with the density peak-based clustering. The proposed detector is evaluated on both simulated and real hyperspectral data sets and the experimental results demonstrate its superiority compared with several traditional and state-of-the-art anomaly detectors.

Anomaly detection is of great importance among hyperspectral applications, which aims at locating targets that are spectrally different from their surrounding background. A variety of anomaly detection methods have been proposed in the past. However, most of them fail to take the high spectral correlations of all the pixels into consideration. Low-rank representation (LRR) has drawn a great deal of interest in recent years, as a promising model to exploit the intrinsic low-rank property of hyperspectral images. Nevertheless, the original LRR model only analyzes the spectral signatures without taking advantage of the valuable spatial information in hyperspectral images. Furthermore, it has been shown that the local geometrical information of the hyperspectral data is also important for discrimination between the anomalies and background pixels. In this article, we incorporate the graph regularization and total variation (TV) regularization into the LRR formulation and propose a novel anomaly detection method based on graph and TV regularized LRR (GTVLRR) model, to preserve the local geometrical structure and spatial relationships in hyperspectral images. Extensive experiments have been conducted on both simulated and real hyperspectral data sets. The experimental results demonstrate the superiority of the proposed method over conventional and state-of-the-art anomaly detection methods.

This paper presents a novel method for hyperspectral anomaly detection based on total variation and structured dictionary. Generally, a hyperspectral imagery can be modeled as a superposition of two components: background and anomalies. Since each pixel in the background can be well represented by some of the other background pixels and the anomalies to be detected can be approximately represented by some potential anomalous pixels. Therefore, each test pixel can be represented using a structured dictionary consisting of background and potential anomalous pixels. Moreover, considering the spatial homogeneity of natural background and the sparse nature of the anomalies, two regularization terms named total variation and sparisty are imposed in the formulation. The experimental results on simulated and real hyperspectral data sets validated the effectiveness of our proposed method compared to several conventional and state-of-the-art anomaly detection methods.

Hyperspectral image classification with limited training samples has become a hot research topic recently. Though deep convolution neural network shows powerful ability for feature extraction, its good performance often relies on sufficient training data. In this paper, we propose a multitask learning framework based on siamese network and auto-encoder to fully exploit limited labeled samples’ information and obtain discriminative features for classification of hyperspectral images. A low intraclass and high interclass variability of features can be learned by metric learning using our framework. And superpixel-based 3D sample preprocessing is applied to improve the classification accuracy on the hyperspectral images’ boundaries. The experimental results demonstrate that our framework can achieve competitive results compared with the state-of-the-art methods.

The imbalance between high dimensionality and limited labeled samples has been a great challenge for classification task of hyperspectral images (HSIs). In this paper, a novel semi-supervised classification method for HSIs is proposed. This method contains two major parts: representation using spatial-spectral graph model and graph convolutional networks (GCN) using edge-conditioned convolution. For the proposed method, spatial-spectral information is considered simultaneously during the process of graph construction, and then GCN is used to extract the feature from input data and learn their topology relationships with edge label involved. Experimental results on multiple hyperspectral datasets with various contexts and resolutions demonstrate that the proposed classifier outperforms several graph-based methods.

The sparse representation-based detection (SRD) algorithm has already shown the effectiveness for hyperspectral target detection (TD) recently. However, SRD does not utilize spatial information of hyperspectral imagery (HSI). In this letter, a novel tensor SRD (TSRD) algorithm is proposed to take jointly the spatial and the spectral information of HSI into account. TSRD extends each atom of both the target and the background dictionaries into a third-order tensor where the local spatial neighborhood information can be well preserved. It not only possesses the advantages of SRD that no assumptions about the target and background distributions are required and spectral variability can be considered but also exploits the spatial information of HSI to further increase the accuracy of TD. The experimental results on both synthetic and real hyperspectral data show that the proposed TSRD method outperforms traditional and state-of-the-art TD methods in terms of detection accuracy.

The scarcity of labeled samples has been the main obstacle to the development of scene classification for remote sensing images. To alleviate this problem, the efforts have been dedicated to semisupervised classification which exploits both labeled and unlabeled samples for training classifiers. In this letter, we propose a novel semisupervised method that utilizes the effective residual convolutional neural network (ResNet) to extract preliminary image features. Moreover, the strategy of ensemble learning (EL) is adopted to establish discriminative image representations by exploring the intrinsic information of all available data. Finally, supervised learning is performed for scene classification. To verify the effectiveness of the proposed method, it is further compared with several state-of-the-art feature representation and semisupervised classification approaches. The experimental results show that by combining ResNet features with EL, the proposed method can obtain more effective image representations and achieve superior results.

Traditional target detection (TD) methods for hyperspectral imagery (HSI) suffer from background interference. In this paper, we propose a novel preprocessing method based on tensor principal component analysis (TPCA) to separate the background and target apart. In our approach, HSI is decomposed into the sum of the principal component (PC) part and the residual part, and TD is performed on the latter. TPCA takes spatial and spectral information into account jointly, and treats spatial and spectral information differently, which is in line with HSI physical meanings. Experiments on both synthetic and real data indicate that our TPCA-based method outperforms other feature extraction preprocessing methods in terms of TD results.

A novel method for hyperspectral anomaly detection based on low-rank representation with manifold regularization is proposed in this paper. Usually, a hyperspectral imagery can be modeled as a superposition of two parts: background part with low rank dimensionality and anomaly part described by a sparse matrix. Low-rank representation (LRR) can be used to find the lowest rank representation of all pixels jointly which represents the background part, then the anomaly part is contained in the residual of the original image. To learn a more discriminative representation, we incorporate the manifold regularization term into the original LRR model. An important advantage of the proposed method is that it can utilize the global low rank property and local geometrical structure jointly. The experimental results on both simulated and real hyperspectral datasets validate the effectiveness of the proposed method.

The special characteristic of remote sensing (RS) images being large scale while only low number of labeled samples available in practical applications has been obstacle to the development of RS image classification. In this paper, a novel semi-supervised framework is proposed. The high-capacity convolutional neural networks (CNN) are adopted to extract preliminary image features. The strategy of ensemble learning is then utilized to establish discriminative image representations by exploring intrinsic information of available data. Plain supervised learning is finally performed to obtain classification results. To verify the efficacy of our work, we compare it with mainstream feature representation and semi-supervised approaches. Experimental results show that by utilizing CNN features and ensemble learning, our framework can obtain more effective image representations and achieve superior results compared with other paradigms of semi-supervised classification.

Wavelength dependent nonlinearity is an essential issue in hyperspectral unmixing, which was overlooked in the past. In this paper, a band-wise nonlinear unmixing method is presented. An extended multilinear mixing model is adopted for interpreting different degrees of nonlinear contributions per band. Moreover, regularizers including abundances' sparsity and nonlinear parameters' smoothness are exploited to formulate the optimization problem and obtain better unmixing results. Finally, unmixing is implemented in the scheme of alternating direction method of multipliers. Experimental results on both simulated and real hyperspectral data validate that the proposed method can improve the unmixing accuracy and reveal the change of nonlinearity at each band as well.

Most nonlinear mixture models and unmixing methods in the literature assume implicitly that the degrees of multiple scatterings at each band are the same. However, it is commonly against the practical situation that spectral mixing is intrinsically wavelength dependent, and the nonlinear intensity varies along with bands. In this paper, a band-wise nonlinear unmixing algorithm is proposed to circumvent this drawback. Pixel dependent probability parameters of the recent multilinear mixing model that represent different orders of nonlinear contributions are vectorized. Therefore, each band can get a scalar probability parameter which explicitly corresponds to the nonlinear intensity at that band. Before solving the extended model, abundances’ sparsity and probability parameters’ smoothness are exploited to build two physical constraints. After incorporating them into the objective function as regularization terms, the issue of local minima can be well alleviated to produce better solutions. Finally, alternating direction method of multipliers is applied to solve the constrained optimization problem and implement the nonlinear spectral unmixing. Experiments are further carried out with current model-based simulated data, physical-based synthetic data of virtual vegetated areas, and real hyperspectral remote sensing images, to provide a more reasonable validation for the developed model and algorithm. In comparison with state-of-the-art nonlinear unmixing methods, this method performs better in explaining the band dependent nonlinear mixing effect for improving the unmixing accuracy.

Recently, many nonlinear spectral unmixing algorithms that use various bilinear mixture models (BMMs) have been proposed. However, the high computational complexity and intrinsic collinearity between true endmembers and virtual endmembers considerably decrease these algorithms’ unmixing performances. In this paper, we come up with a novel abundance estimation algorithm based on the BMMs. Motivated by BMMs’ geometric characteristics that are related to collinearity, we conduct a unique nonlinear vertex${p}$to replace all the virtual endmembers. Unlike the virtual endmembers, this vertex${p}$actually works as an additional true endmember that gives affine representations of pixels with other true endmembers. When the pixels’ normalized barycentric coordinates with respect to true endmembers are obtained, they will be directly projected to be their approximate linear mixture components, which removes the collinearity effectively and enables further linear spectral unmixing. After that, based on the analysis of projection bias, two strategies using the projected gradient algorithm and a traditional linear spectral unmixing algorithm, respectively, are provided to correct the bias and estimate more accurate abundances. The experimental results on simulated and real hyperspectral data show that the proposed algorithm performs better compared with both traditional and state-of-the-art spectral unmixing algorithms. Both the unmixing accuracy and speed have been improved.

Change detection in multitemporal hyperspectral images (HSI) can be regarded as a classification task, consisting of two steps: change feature extraction and identification. To extract clean change features from heavily corrupted spectral change vectors (SCV) of multitemporal HSI, this paper proposes a novel spectrally-spatially regularized low-rank and sparse decomposition model (LRSDSS). It exploits the underlying data structure of SCV by decomposing SCV into three components: spatially smoothed low-rank data, sparse outliers and Gaussian noise. The first component maintains clean change features. The second and the third are corruptions to be removed. The experimental results can validate the effectiveness and the efficiency of LRSD_SS.

Nonnegative matrix factorization (NMF) is often used for unsupervised spectral unmixing in recent years. In this paper, a constrained NMF algorithm based on the bilinear mixture models for unsupervised nonlinear spectral unmixing is proposed. By using a distance measure without dimension reduction, data's projection on a group of constructed hyperplanes representing the nonlinearity are obtained so that the linear parts of data can be approximately determined. Further, we adopt NMF incorporated with a minimum distance constraint for unmixing with the hyperplanes being reconstructed repeatedly during the iteration. Experimental results on synthetic and real hyperspectral data indicate that the proposed algorithm has good unmixing performance.

Nonlinear spectral unmixing based on the bilinear mixture models has received much attention recently. In this paper, an abundance estimation algorithm based on the geometric characteristics of bilinear mixture models is proposed. By representing the models' bilinear terms as the linear contribution of an extra vertex that concentrates the common nonlinear mixing effect, solving the complex bilinear mixture models can be converted to doing the simple linear spectral unmixing. Furthermore, a traditional linear spectral unmixing algorithm is adopted to estimate the abundances directly in an iterative way. Experimental results on synthetic and real hyperspectral data show that the proposed algorithm performs better in both unmixing accuracy and computational speed.

Nonnegative Matrix Factorization (NMF) has been applied to hyperspectral unmixing for a few years. To relieve the non-convex problem, different constraints are imposed on NMF. But these constraints are added only on endmember or abundance. Simultaneously imposing constraints on endmember and abundance has not been tried yet. In this paper, we impose constraints on endmember and abundance at the same time in order to take a more comprehensive consideration of the properties of the hyperspectral image data. The constraints consider not only the geometric feature of endmember but also the sparsity and smoothness of abundance. The experimental performances of our method and other state-of-the-art constrained NMF methods are compared and analyzed, proving that our method is better than only imposing constraints on endmember or abundance and can improve the accuracy of hyperspectral unmixing.

Scarcity of labeled samples is the main obstacle for hyperspectral image classification tasks when labeling data is considerably costly and time-consuming in real-world scenarios. To alleviate any underfitting problem that may occur due to lack of training data, semisupervised classification frameworks explore the intrinsic information of unlabeled samples and bridge labeled and unlabeled data. In this letter, we propose a novel framework that learns underlying manifold representation and semisupervised classifier simultaneously. It avoids explicit eigenvector decomposition and directly samples via iterating random walk on the similarity graph, which makes it feasible to implement on huge graphs. To verify the efficacy of embedding the learning process, we compare the proposed method with other dimensionality reduction and manifold-learning-based approaches. Experimental results show that compared to the methods using traditional semisupervised strategies, the graph embedding method gives a better result.

The accuracy of target spectra determines the performances of hyperspectral target detection (TD) algorithms. However, given the inherent spectral variability and subpixel problem in hyperspectral imagery (HSI), the target spectra obtained from a standard spectral library or pixels from images directly are in most cases different from those of the real target spectra, resulting in low detection accuracy. The problem caused by inaccurate prior target information led to recognition of a new hotspot on HSI. In this paper, an adaptive weighted learning method (AWLM) using a self-completed background dictionary (SCBD) is specifically developed to extract the accurate target spectrum for hyperspectral TD. AWLM is derived from the idea of dictionary learning algorithms, learning the specific target spectrum with target-proportion-related adaptive weights. A strategy to construct SCBD is proposed to guarantee the convergence of AWLM to the accurate target spectrum. Utilizing the extracted target spectrum with higher accuracy, conventional TD algorithms can also achieve satisfactory detection results. Experimental results on both simulated and real hyperspectral data demonstrate that the proposed method has an advantage in extracting accurate target spectrum, enabling better and more robust detection results using conventional detectors than state-of-the-art methods that also aim at the problem of inaccurate prior target information of HSI.

Classification has been among the central issues of hyperspectral application. However, due to the well-known Hughes phenomenon, most of the methods suffer from the curse of dimensionality and deeply rely on traditional dimensional reduction like Principle Component Analysis (PCA). In this paper, combining spatial and spectral information jointly, we propose a novel deep classification framework. It consists of two parts: graph-based spatial fusion and Convolutional Neural Network (CNN). Spatial fusion acts as a pre-training stage that extracts spatial-spectral features from high-order data. CNN learns and infers spectrum efficiently from fused input via deep hierarchy with convolutional and pooling layers, thus forming a relationship between spectral-spatial features and class distribution. Experiment results show that the performance of the proposed classifier is competitive enough with other pixel-wise classifiers.

To combat the well-known Hughes phenomenon occurred in hyperspectral classification, most of the previous works adopt dimensionality reduction or manifold learning technique before supervised learning. While in this paper, we propose a different scheme: First, we design a pixel-wise classifier based on Convolutional Neural Network that could directly mapping observed spectrum to class distribution. Then, we conduct superpixel segmentation on the prediction map that learned by previous model and output the final classification results by spatial and spectral factors jointly. Varied from other deep learning method, our classification framework learns and infers spectrum efficiently via deep hierarchy with convolutional and pooling layers, thus forming a direct relationship between high-order data and class distribution. Moreover, superpixel segmentation helps further boost the accuracy of the classification by combining the spatial information. In experimental studies, multiple hyperspectral datasets with various context and spatial resolution are used to validate the proposed method. The experimental results show that the proposed method is efficient and competitive in practical uses.

Change detection for multitemporal hyperspectral images (HSIs) involves two major steps: change feature extraction and classification. For the first part, conventional methods mostly consider spectral features but neglect spatial patterns. Since multitemporal HSIs consist of four dimensions (one for time, one for spectral domain and two for spatial domain), we propose using 4-dimensional Higher Order Singular Value Decomposition (4D-HOSVD) based on tensor algebra to capture the details in all the dimensions simultaneously and thus producing comprehensive change features. To emphasize on the effectiveness of the change feature extraction method, this paper reduces the change classification to a simple binary problem: a pixel is either changed or unchanged. Experimental results show that 4D-HOSVD can outperform its matrix counterpart, Principal Component Analysis (PCA), as well as some other widely adopted method.

There are two great challenges for classification of hyperspectral images (HSIs): lack in prior knowledge and serious internal-class variability. To address the issues, we propose a novel semisupervised method based on affinity scoring (AS). It can harness the fuzzy state of the contributions of spectral and spatial features to classification. The method consists of three major steps: over-segmentation, semisupervised classification and modification. First, superpixels are generated to maintain local class consistency, which can balance spectral variability. Then unlabeled samples are classified by AS in an iterative manner, whereas precious labeled samples are made most use of. Finally, AS is adopted again to refine the classification map, which further exploits spatial smoothness in HSIs. Experiments show that the proposed method can largely outperform several state-of-the-art classifiers.

Sparse representation has been introduced to tackle the target detection problem in hyperspectral imagery. While using windows to build the sparse dictionary, there exists target contamination problem. In our approach, we utilize a learning method based on convex optimization to build a dictionary for sparse target detection. Through its application, prior information such as the size of windows can be spared, while considerably reducing the occurrence of contamination. To verify the efficacy of using the learned dictionary, the dictionary built through the dual-window method is used as a comparison and two sparse target detection methods are employed afterward. Experimental results show that, by using the learned dictionary, a better result is obtained compared to the methods using traditional dual-window background dictionary.

Semi supervised classification has become popular, since it can make use of a limited amount of prior knowledge in hyperspectral images. However, spectral internal-class variability adds a great challenge to the task. To address these issues, we propose a novel semi supervised spectral-spatial classification method based on affinity scoring (AS) (SCAS). Adapted from fuzzy logic, AS exploits spectral and spatial features with their fuzzy contributions to classification by weighing on three factors: local class consistency, spectral similarity, and prior knowledge. SCAS consists of three main steps: oversegmentation, semi supervised classification, and modification. The first step generates super pixels and uses them to maintain local class consistency. The second and third steps employ AS to classify the super pixels and refine the classified map, respectively. Experiments show that the proposed method can outperform some classic methods and state-of-the-art classifiers.

The geometrical features of airport line segments are seldom used by traditional methods for airport detection in panchromatic remote sensing images. This letter presents a novel method based on both bottom-up (BU) saliency and top-down saliency. Noticing that airport runways have features of vicinity and parallelity and that their lengths are among a certain range, we introduce the concept of near parallelity for the first time and treat it as prior knowledge that can fully exploit the geometrical relationship of airport runways. Meanwhile, a simplified graph-based visual saliency model is used to extract the BU saliency. Two-way results are combined, and candidate regions can be derived from it. Finally, a scale-invariant feature transform and a support vector machine are used to determine whether the regions contain airports or not. The proposed method is tested on an image data set composed of different kinds of airports. The experimental results show that the method outperforms other state-of-the-art models in terms of speed, the detection rate, and the false-alarm rate. In addition, the method is more robust to a complex background than the other methods.

Hyperspectral unmixing is an important issue in hyperspectral image processing. In this paper, we transform the unmixing problem into a constrained nonlinear least squares (CNLS) problem by introducing the abundance sum-to-one constraint, abundance nonnegative constraint, and bound constraints on nonlinearity parameters. The new CNLS-based algorithms assume that the mixing mechanism of each observed pixel can be described by two forms. One is a sum of linear mixtures of endmember spectra and nonlinear variations in reflectance, and the other is a joint mixture resulting from the linearity and nonlinearity in hyperspectral data. For the former, an alternating iterative optimization algorithm is developed to solve the problem of CNLS. As for the latter, the structured total least squares optimization approach is used to obtain the abundance vectors and nonlinearity parameters simultaneously. Current mixing models can be interpreted by either or both of these two mechanisms. A comparative analysis based on Monte Carlo simulations and real data experiments is conducted to evaluate the proposed algorithms and five other state-of-the-art algorithms. Experimental results show that the proposed algorithms give outstanding performance of hyperspectral nonlinear unmixing for both synthetic data and real hyperspectral images, as satisfactory accuracy in term of abundance fractions and low computational complexity are observed.

In many applications, it is imperative to maintain high spectral and spatial resolution of remote sensing images. This letter addresses the issue by fusing low-spatial-resolution hyperspectral images (HSIs) and high-spatial-resolution multispectral images (MSIs) of the same scene collected by the coupled sensors and, thus, present a novel framework that generalizes well-established pan-sharpening algorithms. The main steps of the framework are dividing the spectrum of HSIs into several regions and fusing HSIs and MSIs in each region by the chosen pan-sharpening algorithm. Ratio image-based spectral resampling (RIBSR) is used to interpolate the missing data so that every region is covered by a multispectral band. Therefore, the framework allows most of pan-sharpening algorithms to be extended to HSI and MSI fusion. Synthetic data in accordance with sensor reality are used to test specific methods derived within the framework. Experimental results show that the proposed methods excel the state-of-the-art methods in terms of simplicity, feasibility, efficiency, and effectiveness.

In recent years, dimensionality reduction (DR) and classification have become important issues of hyperspectral image analysis. In this paper, we propose a new spatial–spectral similarity measure, which maps the distances between two image patches in hyperspectral images. Including spatial information by using the spatial neighbors, the proposed similarity measure is based on the fact that the observed pixels in the images are spatially related, and the meaningful features can be extracted from both the spectral and spatial domains. First, the new similarity measure can effectively exploit the rich spectral and spatial structures of data, thus improving the original$k$-nearest neighbor ($k$NN) classification methods. Second, the new similarity measure can be incorporated into existing DR methods including linear or nonlinear techniques. With the merits of the proposed similarity measure, the modified DR methods become effective in dealing with the redundancy resulting from spectral signature as well as the spatial relation among pixels. A comparative study and analysis based on classification experiments using five real hyperspectral data sets, which were acquired by different instruments, is conducted to evaluate the proposed similarity measure. The experimental results demonstrate that the proposed measure is promising for combining spectral and spatial information when applied to DR and classification of hyperspectral data sets.

Under the linear spectral mixture model, hyperspectral unmixing can be considered as a convex geometry problem, in which the endmembers are located in the vertices of simplex enclosing the hyperspectral data set and the barycentric coordinates of observation pixels with respect to the simplex correspond to the abundances of endmembers. Based on distance geometry theory, in this paper we propose a new approach for abundance estimation of mixed pixels in hyperspectral images. With the endmember signatures, which is known a priori or can be obtained from the endmember extraction algorithms, the proposed method automatically estimates the abundances of endmembers at each pixel using convex geometry concepts and distance geometry constraints. In the algorithm, denoting the pairwise distances with Cayley-Menger matrix makes it easy to calculate the barycentric coordinates of the observation pixels. Another characteristic of this algorithm is that the optimal estimated points of observation pixels as well as the least distortion in geometric structure of original data set can be obtained with the distance geometry constraint. Simultaneously, the use of barycenter of simplex builds an accurate and efficient method to estimate endmembers with zero abundance and, as a result, the subsimplex containing the estimated points is obtained. A comparative study and analysis based on Monte Carlo simulations and real data experiments is conducted among the proposed algorithm and three state-of-the-art algorithms: fully constrained least squares (FCLS), FCLS computed using constrained sparse unmixing by variable splitting and augmented Lagrangian, and simplex-projection unmixing (SPU). The experimental results show that the proposed algorithm always provides the best unmixing accuracy and when the number of endmembers is not very large the algorithm has a lower computational complexity.

This paper proposes a new similarity measure to integrate the spectral and spatial-contextual information in the hyperspectral imagery into the manifold learning methods. Including spatial information using the spatial neighbor, the proposed similarity measure is based on the fact that the observation pixels in the hyperspectral imagery are spatially related and relevant information can be extracted from both the spectral and spatial domains. The proposed nonlinear dimensionality reduction techniques based on the new similarity measure can effectively deal with the nonlinearity in the real hyperspectral data as well as the spatial relation among pixels, leading to a more meaningful and manageable representation of original high-dimensional data set with reduced dimensionality. The results from the real hyperspectral image experiments denote that the proposed algorithms significantly increase the classification accuracy for the hyperspectral images compared with other spectral based dimensionality reduction methods.

Hyperspectral unmixing is an important issue to analyze hyperspectral data. Based on the present mixing models, this paper proposes a new nonlinear unmixing framework for hyperspectral imagery. The proposed framework transforms the hyperspectral unmixing problem to a constrained nonlinear least squares problem by introducing the abundance nonnegative constraint, abundance sum-to-one constraint and the bound constraints of nonlinear parameters. Accordingly, an alternating iterative optimization algorithm is developed to solve the arising nonlinear least squares problem. The method decomposes the nonlinear unmixing problem into two sub-problems, which obtain alternately the abundance vectors and nonlinear parameters of the observation pixels. The experimental results on synthetic and real hyperspectral dataset demonstrate that the proposed algorithm can effectively overcome the inherent limitations of the linear mixing model. Meanwhile, the proposed algorithm performs well for noisy data, and can also be used as an effective technique for the nonlinear unmixing of hyperspectral imagery.

In recent years, band selection is becoming a popular approach to reduce the dimensionality of hyperspectral data while preserving the desired information for target detection and classification analysis. This letter presents a new method for unsupervised band selection by transforming the hyperspectral data into complex networks. By analyzing the networks' topological feature corresponding to each band, one can easily evaluate the statistical characteristics and intrinsic properties of the signals. The proposed method searches for the network set which is most qualified for demarcating and identifying different substance signatures, and then, the network set's corresponding bands are regarded as the descried output results. This network measure is a new criterion for band selection. Experimental results demonstrate that the proposed method can acquire satisfactory results when compared with traditional methods.

This paper addressed the evaluation of Temperature Independent Spectral Indices of Emissivity (TISIE) in Land Surface Emissivity (LSE) retrievals over an eastern China area (110°E–125°E, 35°N–50°N) using Radiative Transfer Modeling (RTM) experiments with the products (MOD11C1, MOD11_L2 and MOD07) of the MODerate resolution Imaging Spectroradiometer (MODIS) on Terra in Sept. of 2007. The results show that the LSE errors due to the instrumental noises can be neglected, whereas the LSE errors due to the method and the uncertainties of Total Precipitable Water (TPW) are channel-dependent and region-dependent, and in general, they are so large that they cannot be neglected in both middle infrared and thermal infrared channels.

This paper proposed a new approach to estimate the abundance of each endmember at each pixel using distance geometry concepts and distance geometry constraints. It improves current hyperspectral unmixing algorithms in several aspects. Firstly, denoting the distance relationship with Cayley-Menger matrix makes it easy to calculate the barycentric coordinates of observation pixels, and the computation is independent of number of bands. Secondly, by the distance geometry constraint, the geometric structure of dataset is considered to obtain the optimal result with least geometric deformation. The synthetic and real data experimental results demonstrate that this algorithm is a fast and accurate algorithm for the hyperspectral unmixing.

The hyperspectral bands are contiguous and highly correlated spectral bands. Band selection is often used to reduce the computational complexity for hyperspectral images. We proposed a new method for unsupervised band selection by using complex network to represent the spectral bands. The method completes the task with the objective of preserving the maximal information from original data in the selected bands. Both the divergences and connections between each hyperspectral band can be revealed from the topological characteristics of the generated network. We use the network topology as the criterion to identify the bands, and select the bands that can form the most approximate network comparing to the network of the original data. Experimental results demonstrate that, compared with traditional methods, the proposed algorithm can obtain accurate results with clear physical meaning and simple process.

In the linear unmixing of hyperspectral images, the observation pixels form a simplex whose vertices correspond to the endmembers, hence finding the endmembers is equivalent to extracting these vertices. A common technique for determining vertices is to analyze the simplex volume, but it usually has a high computational complexity, resulting from the exhaustive searching of volume in the large hyperspectral data. This problem limits the practicability and real-time application. In this paper, we utilize triangular factorization (TF) to calculate the volume, deducing a method named simplex volume analysis based on TF (SVATF). It requires just one comparison through the data to succeed in finding the global optimal solution for all the endmembers, thus improving the searching efficiency. Dimensionality reduction transformation is not necessary, which is another advantage of this method. Moreover, since TF is a broad conception including different methods, SVATF is a framework including various implementations. Based on TF, we also propose a fast learning algorithm named abundance quantification based on TF to estimate the abundances, which further saves the computation by utilizing the intermediate values involved in SVATF. The abundance estimation method can rectify possible errors in the given endmembers by utilizing two important constraints (abundance nonnegative constraint and abundance sum-to-one constraint) of the linear mixture model, so it is useful for the imagery without pure pixels. Experimental results on synthetic and real hyperspectral data demonstrate that the proposed methods can obtain accurate results with much lower computational complexity, with respect to other state-of-the-art methods.

Endmember extraction is a process to identify the spectra of materials from the hyperspectral scene. This paper presents a framework for endmember extraction by exploiting the ideas that: the endmembers are the vertices of the simplex, and the calculation of simplex volume can be simplified by triangular factorization. Triangular factorization is a broad conception including many methods, so the proposed framework is a group of methods including different implementations. Experimental results on both synthetic and real hyperspectral data demonstrate that the proposed algorithm can obtain the results with better accuracy and much lower complexity, comparing to other state-of-the-art approaches.

Nonnegative matrix factorization (NMF) has been recently applied to solve the hyperspectral unmixing problem because it ensures nonnegativity and needs no assumption for the presence of pure pixels. However, the algorithm has a large amount of local minima due to the obvious nonconvexity of the objective function. In order to improve its performance, auxiliary constraints can be introduced into the algorithm. In this paper, we propose a new approach named abundance separation and smoothness constrained NMF by introducing two constraints, namely, abundance separation and smoothness, into the NMF algorithm. These constraints are based on two properties of hyperspectral imagery. First, usually, every ground object presents dominance in a specific region of the entire image scene and the correlation is weak between different endmembers. Second, moving through various regions, ground objects usually vary slowly and abrupt changes rarely appear. We also propose a learning algorithm to further improve the performance of our method, from which the auxiliary constraints are removed at an appropriate time. The proposed algorithm retains all the advantages of NMF and effectively overcomes the shortcoming of local minima at the same time. Experimental results based on synthetic and real hyperspectral data show the superiority of the proposed algorithm with respect to other state-of-the-art approaches.

In recent years, independent component analysis (ICA) has been applied to unmix the hyperspectral data since it can perform without the prior knowledge of ground objects. The traditional ICA algorithm regards the extracted independent components as unmixing results, which is not reasonable for hyperspectral imagery, because different endmembers are not actually independent from each other. In order to solve this problem, a new approach, named as constrained ICA, is proposed, in which we consider “uncorrelation” instead of “independence.” Two constraints of the hyperspectral data (the abundance nonnegative and abundance sum-to-one constraints) are introduced to the ICA, changing its objective function based on independence assumption. Furthermore, we develop a technique, called as adaptive abundance modeling, to characterize the statistical distribution of the data. The model is automatically constructed according to the given data, which can encourage the algorithm that is applicable to various hyperspectral images with different statistical characteristics. The experimental results on both simulated and real hyperspectral data demonstrate that the proposed approach can obtain more accurate results with respect to existing algorithms. As an algorithm with no need of prior spectral knowledge, our method provides an effective solution for the blind unmixing of the hyperspectral data.

Traditional Nonnegative Matrix Factorization (NMF) algorithm is sensitive to the initial value when being applied to hyperspectral unmixing, because of the local minima in the objective function. In order to solve the problem, two constraints of abundance separation and smoothness are introduced into the NMF algorithm. The proposed algorithm retains the advantages of NMF, and effectively overcomes the shortcoming of local minima at the same time. Experimental results on simulated and real hyperspectral data demonstrate that the proposed approach can overcome the shortcoming of local minima, and obtain better results with respect to other state-of-art approaches. Meanwhile, the algorithm performs well for noisy data, and can also be used for the unmixing of hyperspectral data in which pure pixels do not exist.

In hyperspectral unmixing, endmember signals are not independent with each other, restricting the application of independent component analysis (ICA). We present a new algorithm to overcome this problem. By introducing abundance nonnegative and abundance sum-to-one constraints into objective function of ICA, the goal of our method is changed from “independence” to “uncorrelation”. We also develop an abundance modeling technique to describe the statistical distribution of hyperspectral data. The modeling approach is capable of self-adaptation, and can be applied to various images with different characteristics. Experimental results on both simulated and real hyperspectral data demonstrate that the proposed approach can obtain accurate results. As an algorithm with no need of spectral prior knowledge, our method provides an effective technique for hyperspectral unmixing.

Traditional spectral mixture analysis assumes that each endmember must have a constant spectral signature. However, endmember spectral variability always exists in practical situations, which results in reducing the accuracy of the decomposition of mixed pixels. In order to solve this problem, this letter proposes a new method based on Fisher discriminant null space (FDNS) for decomposition of mixed pixels in hyperspectral imagery. The FDNS searches a linear transformation of the spectra, which makes those endmember spectra to have no variability inside each endmember group but large differences among different endmember groups. Therefore, the negative impact caused by endmember spectral variability on unmixing accuracy can be decreased to a large extent by using the transformed spectra. Experimental results of both simulated and real hyperspectral images demonstrate that the proposed algorithm has a high accuracy for the decomposition of mixed pixels in hyperspectral imagery.

Compressive sampling is a novel framework in signal acquisition and reconstruction, which achieves sub-Nyquist sampling by exploiting the sparse nature of most signals of interest. In this letter, we propose a saliency-based compressive sampling scheme for image signals. The key idea is to exploit the saliency information of images, and allocate more sensing resources to salient regions but fewer to nonsalient regions. The scheme takes human visual attention into consideration because human vision would pay more attention to salient regions. Simulation results on natural images show that the proposed scheme improves the reconstructed image quality considerably compared to the case when saliency information is not used.

The N-FINDR algorithm has been widely used in hyperspectral image analysis for endmember extraction due to its simplicity and effectiveness. However, there are several disadvantages of implementing the N-FINDR. This letter proposes an algorithm for decomposition of mixed pixels. It improves the N-FINDR in several aspects. First, an iterative Gram-Schmidt orthogonalization is applied in the endmember searching process to replace the matrix determinant calculation used in N-FINDR, which makes this algorithm run very fast and can also guarantee the stability of its final results. Second, with the set of orthogonal bases obtained by the Gram-Schmidt orthogonalization, the algorithm can also help to estimate the proper number of endmembers and unmix the original images by itself. In addition, unlike the N-FINDR, a dimensionality reduction transform is not necessary in this algorithm. Experimental results of both simulated images and practical remote sensing images demonstrate that this algorithm is a fast and accurate algorithm for the decomposition of mixed pixels.

How to decompose mixed pixels effectively and precisely for remote sensing images is a critical issue for the quantitative remote sensing research. In this paper we propose a new method for decomposition of mixed pixels of multispectral or hyperspectral remote sensing images. The proposed method introduces the algorithm of Bayesian Self-Organizing Map (BSOM) into the problem of the decomposition of mixed pixels. It estimates Gaussian parameters by minimizing the Kullback- Leibler information metric, and finishes the unmixing with Gaussian Mixture Model (GMM). In order to obtain a high unmixing precision, we need to extend the range of Gaussian distributions, and thus propose the 3 σ variances adjustment method to solve this problem. In addition, the used unmixing model satisfies two constraints which are demanded for the problem of the decomposition of mixed pixels automatically: abundances non-negative constraint (ANC) and abundances summed-to-one constraint (ASC). Experimental results on simulated and practical remote sensing images demonstrate that the proposed method can get good unmixing results for the decomposition of mixed pixels, and is more robust to noise than other methods.

The simplex-based methods are a kind of the most important and widely used methods for the decomposition of mixed pixels in multispectral and hyperspectral remote sensing images, which need a strong basic hypothesis that there is at least one pure pixel for every endmember existing in the images. N- FINDR is of typical sense in this kind of methods, and it also needs this basic hypothesis. Unfortunately, the precision for the decomposition of mixed pixels will be seriously influenced if this hypothesis cannot be met in practice. This paper presents a new scheme based on Nonnegative Matrix Factorization (NMF) to solve this problem. In addition, some appropriate constrains are introduced into NMF for the decomposition of mixed pixels. Experimental results obtained from both artificial simulated and real-world remote sensing data demonstrate that the proposed scheme for decomposition of mixed pixels has excellent analytical performance.

A new algorithm for decomposition of mixed pixels based on orthogonal bases of data space is proposed in this paper. It extracts endmembers sequentially by adding a new vertex of the simplex with the maximum volume every time. It can avoid the dilemma in traditional simplex-based endmember extraction algorithms such as N-FINDR that it generally produces different sets of final endmembers if different initial conditions are used. Meanwhile, at each step of the searching process, the calculation of a simplex volume based on determinant of a matrix is replaced by the one based on the norm products of a set of orthogonal bases, which results in a tremendous reduction of computational complexity. Moreover, with this set of orthogonal bases, the proposed algorithm can also determine the proper number of endmembers and finish the unmixing of the original images which cannot be done by the traditional endmember extraction algorithms. Experimental results of practical remote sensing images demonstrate that the algorithm proposed in this paper is a fast and accurate algorithm for the decomposition of mixed pixels.

In this paper, a hypercomplex principle component weighted approach to multiplespectral and panchromatic images fusions is presented. Since it takes the vectorial property of RGB components into account, there is not the distortion of visible spectrums in the images fused by the proposed method. The evaluation results from the various methods representing the state of the art ones also show that the fusion results of presented approach is better than those of IHS(hue-intensity-saturation), PCA (principle component analysis) and wavelet-based methods.

Due to interference caused by ground transmitters, so-called ghost noise occurs on images obtained from synthetic aperture radar (SAR). This letter proposes a scheme based on the wavelet packet transform to remove this ghost noise from the SAR image. The wavelet packet transform is employed to decompose the SAR image with the ghost noise both in the spatial and in the spatial frequency domain. To automatically detect and remove the ghost noise, an energy detection method is developed in this letter. The performance of the proposed scheme is demonstrated by some experimental results on real SAR images. In addition, we point out that the proposed scheme is also useful for other applications of image processing and analysis.