Multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution.

Hyperspectral (HS) pansharpening aims at fusing an observed HS image with a panchromatic (PAN) image, to produce an image with the high spectral resolution of the former and the high spatial resolution of the latter. Most of the existing convolutional neural networks (CNNs)-based pansharpening methods reconstruct the desired high-resolution image from the encoded low-resolution (LR) representation. However, the encoded LR representation captures semantic information of the image and is inadequate in reconstructing fine details. How to effectively extract high-resolution and LR representations for high-resolution image reconstruction is the main objective of this article. In this article, we propose a feature pyramid fusion network (FPFNet) for pansharpening, which permits the network to extract multiresolution representations from PAN and HS images in two branches. The PAN branch starts from the high-resolution stream that maintains the spatial resolution of the PAN image and gradually adds LR streams in parallel. The structure of the HS branch remains highly consistent with that of the PAN branch, but starts with the LR stream and gradually adds high-resolution streams. The representations with corresponding resolutions of PAN and HS branches are fused and gradually upsampled in a coarse to fine manner to reconstruct the high-resolution HS image. Experimental results on three datasets demonstrate the significant superiority of the proposed FPFNet over the state-of-the-art methods in terms of both qualitative and quantitative comparisons.

Accurately distinguishing between background and anomalous objects within hyperspectral images poses a significant challenge. The primary obstacle lies in the inadequate modeling of prior knowledge, leading to a performance bottleneck in hyperspectral anomaly detection (HAD). In response to this challenge, we put forth a groundbreaking coupling paradigm that combines model-driven low-rank representation (LRR) methods with data-driven deep learning techniques by learning disentangled priors (LDP). LDP seeks to capture complete priors for effectively modeling the background, thereby extracting anomalies from hyperspectral images more accurately. LDP follows a model-driven deep unfolding architecture, where the prior knowledge is separated into the explicit low-rank prior formulated by expert knowledge and implicit learnable priors by means of deep networks. The internal relationships between explicit and implicit priors within LDP are elegantly modeled through a skip residual connection. Furthermore, we provide a mathematical proof of the convergence of our proposed model. Our experiments, conducted on multiple widely recognized datasets, demonstrate that LDP surpasses most of the current advanced HAD techniques, exceling in both detection performance and generalization capability.

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.

Most hyperspectral image (HSI) classification methods rely on square patch sampling to incorporate spatial information, thereby facilitating the label prediction of the center pixel. However, square patch sampling introduces numerous heterogeneous pixels, which could distort the label prediction of center pixel. Moreover, it generates fixed training patch sample for each center pixel, hampering the performance of transformer-based models requiring a large number of training data. To address the above problems, we proposed Center Attention Transformer (CAT) with stratified spatial-spectral token generated by superpixel sampling for HSI classification. Firstly, to mitigate the inference of heterogeneous pixels, we propose Sampling From Superpixel Region mechanism to generate purer image cubes than traditional square neighborhood. Secondly, to expand the training data for transformer, we propose Multiple Stratified Random Sampling mechanism, which generates ample training samples without introducing additional labels. Finally, to more effectively extract information from the sampled patch tokens, we propose Spatial Spectral Token Generation mechanism and Center Attention Transformer structure with Gaussian Positional Embedding. This framework can extract long-range correlations of spectral information and pay more attention on the center pixel in spatial dimension. Experimental results on three HSI datasets demonstrate the performance of our proposed method CAT outperforms several state-of-the-art methods. The code of this work is available at https://github.com/fengjiaqi927/CAT-Center_Attention_Transformer.

Recently, few-shot learning based on meta-learning has shown great potential in hyperspectral image classification (HSIC) due to its excellent adaptability to limited training samples. Despite achieving promising results, the existing methods ignore the interaction between the source domain (with abundant-labeled base-class samples) and the target domain (with few-labeled novel-class samples), as well as between the support set and the query set. This issue makes the resulting model usually biased toward the source domain and not robust to the sample variance of novel classes, posing a bottleneck to the improvement of HSIC performance. To overcome these limitations, we propose a flexible and effective distribution-aware and class-adaptive aggregation (DA-CAA) method for few-shot HSIC by transferring the class-level distribution information learned from the base classes to the novel classes. Specifically, we first employ a variational autoencoder (VAE), which is pretrained on abundant-labeled base-class samples, to encode the support set samples as class distributions. Subsequently, we sample class-level features from the learned distribution and adaptively aggregate them with sample-specific query features. This operation not only enhances cross-domain information interaction in a distribution-learning manner, but also ensures that the aggregated features across classes inherit both class-level and sample-specific information. Our proposed class-adaptive aggregation (CAA) encourages complementary fusion of features from all classes, which is beneficial for reducing class confusion. Experiments on four benchmark datasets demonstrate the effectiveness and flexibility of our approach.

Current hyperspectral anomaly detection (HAD) benchmark datasets suffer from low resolution, simple background, and small size of the anomalies. These factors also limit the performance of the well-known low-rank representation (LRR) models in terms of robustness on the separation of background and target features and the reliance on manual parameter selection. To this end, we build a new HAD benchmark dataset for improving the robustness in complex scenarios, AIR-HAD for short, and propose an interpretable network with deep unfolding a binary subspace learning, named LRR-Net+, which is capable of spectrally decoupling the background structure and object properties in a more generalized fashion and eliminating the bias introduced by vital interference targets simultaneously. In addition, LRR-Net+ integrates the solution process of the alternating direction method of multipliers (ADMM) optimizer with the deep network, guiding its search process and imparting a level of interpretability to parameter optimization. Additionally, the integration of physical models with DL techniques eliminates the need for manual parameter tuning. The manually tuned parameters are seamlessly transformed into trainable parameters for deep neural networks, facilitating a more efficient and automated optimization process. Extensive experiments conducted on the AIR-HAD dataset show the superiority of our LRR-Net+ in terms of detection performance and generalization ability, compared to top-performing competitors. Furthermore, our AIR-HAD benchmark datasets will be made available freely and openly athttps://github.com/danfenghong/IEEE_TGRS_LRR-Net.

Weakly supervised semantic segmentation methods can effectively alleviate the problem of high cost and difficult access to annotation in traditional methods. Among these approaches, point annotated semantic label not only offers a more affordable option but also provides accurate location and category information, playing an indispensable role in current research. However, point annotation labeling encounters challenges such as missing global and texture information, and limiting segmentation accuracy and efficiency while being susceptible to noise interference. For the above problems, a weakly supervised remote sensing image classification framework based on point annotated semantic label is proposed, which consists of three components: data augmentation, Pixel-Net, and iterative superpixel-based sample expansion (ISSE). First, the data augmentation method is used to generate a sufficient number of training samples. Subsequently, the weakly supervised network Pixel-Net is trained using point annotated semantic labels. Pixel-Net incorporates traditional image processing techniques such as edge detection and blurring into deep learning, enabling effective learning of edge and spectral semantic details while reducing the impact of noise on classification results. Finally, ISSE leverages contextual information from superpixels and pseudo-labels to enrich the valuable information in weakly supervised labels, thereby improving the model’s classification performance. In the experiments, existing semantic segmentation methods and Pixel-Net are evaluated on the Vaihingen and Zurich Summer datasets, and the effectiveness of ISSE is verified. The results show that Pixel-Net achieves the best segmentation accuracy on both datasets, while ISSE can effectively utilize the existing point annotation labels to mitigate the effect of noise and thus improve the accuracy of weakly supervised semantic segmentation.

This paper proposes a novel uncertainty-adjusted label transition (UALT) method for weakly supervised solar panel mapping (WS-SPM) in aerial Images. In weakly supervised learning (WSL), the noisy nature of pseudo labels (PLs) often leads to poor model performance. To address this problem, we formulate the task as a label-noise learning problem and build a statistically consistent mapping model by estimating the instance-dependent transition matrix (IDTM). We propose to estimate the IDTM with a parameterized label transition network describing the relationship between the latent clean labels and noisy PLs. A trace regularizer is employed to impose constraints on the form of IDTM for its stability. To further reduce the estimation difficulty of IDTM, we incorporate uncertainty estimation to first improve the accuracy of noisy dataset distillation and then mitigate the negative impacts of falsely distilled examples with an uncertainty-adjusted re-weighting strategy. Extensive experiments and ablation studies on two challenging aerial data sets support the validity of the proposed UALT.

With the development of hyperspectral sensors, accessible hyperspectral images (HSIs) are increasing, and pixel-oriented classification has attracted much attention. Recently, graph convolutional networks (GCNs) have been proposed to process graph-structured data in non-Euclidean domains and have been employed in HSI classification. But most methods based on GCN are hard to sufficiently exploit information of ground objects due to feature aggregation. To solve this issue, in this article, we proposed a graph-in-graph (GiG) model and a related GiG convolutional network (GiGCN) for HSI classification from a superpixel viewpoint. The GiG representation covers information inside and outside superpixels, respectively, corresponding to the local and global characteristics of ground objects. Concretely, after segmenting HSI into disjoint superpixels, each one is converted to an internal graph. Meanwhile, an external graph is constructed according to the spatial adjacent relationships among superpixels. Significantly, each node in the external graph embeds a corresponding internal graph, forming the so-called GiG structure. Then, GiGCN composed of internal and External graph convolution (EGC) is designed to extract hierarchical features and integrate them into multiple scales, improving the discriminability of GiGCN. Ensemble learning is incorporated to further boost the robustness of GiGCN. It is worth noting that we are the first to propose the GiG framework from the superpixel point and the GiGCN scheme for HSI classification. Experiment results on four benchmark datasets demonstrate that our proposed method is effective and feasible for HSI classification with limited labeled samples. For study replication, the code developed for this study is available at https://github.com/ShuGuoJ/GiGCN.git.

In recent years, convolutional neural networks (CNNs)-based methods achieve cracking performance on hyperspectral image (HSI) classification tasks, due to its hierarchical structure and strong nonlinear fitting capacity. Most of them, however, are supervised approaches that need a large number of labeled data to train them. Conventional convolution kernels are fixed shape of rectangular with fixed sizes, which are good at capturing short-range relations between pixels within HSIs but ignore the long-range context within HSIs, limiting their performance. To overcome the limitations mentioned above, we present a dynamic multiscale graph convolutional network (GCN) classifier (DMSGer). DMSGer first constructs a relatively small graph at region-level based on a superpixel segmentation algorithm and metric-learning. A dynamic pixel-level feature update strategy is then applied to the region-level adjacency matrix, which can help DMSGer learn the pixel representation dynamically. Finally, to deeply understand the complex contents within HSIs, our model is expanded into a multiscale version. On the one hand, by introducing graph learning theory, DMSGer accomplishes HSI classification tasks in a semi-supervised manner, relieving the pressure of collecting abundant labeled samples. Superpixels are generally in irregular shapes and sizes which can group only similar pixels in a neighborhood. On the other hand, based on the proposed dynamic-GCN, the pixel-level and region-level information can be captured simultaneously in one graph convolution layer such that the classification results can be improved. Also, due to the proper multiscale expansion, more helpful information can be captured from HSIs. Extensive experiments were conducted on four public HSIs, and the promising results illustrate that our DMSGer is robust in classifying HSIs. Our source codes are available at https://github.com/TangXu-Group/DMSGer.

In order to effectively utilize spectral and object spatial information to improve tracking performance, we design an Hyperspectral Video (HSV) tracker, namely DSP-Net, to integrate the various prior information. The gradient difference between spectral vectors is explored to develop a clustering technique. The approach generates a binary mask containing target spectral information and appearance clues. A feature cache is introduced to store historical information. Additionally, the channel shift operation is used on the timing to capture the trajectory clues of the target. With the help of the non-local mechanism, the trajectory clues, appearance clues and spectral information of the target are finally integrated, using the designed spectral-spatial joint perception module to enhance the expression of the target. Experimental results show that DSP-Net outperforms state-of-the-art HSV trackers on existing dataset.

Due to the cubic structure of a hyperspectral image (HSI), how to characterize its spectral and spatial properties in three dimensions is challenging. Conventional spectral-spatial methods usually extract spectral and spatial information separately, ignoring their intrinsic correlations. Recently, some 3D feature extraction methods are developed for the extraction of spectral and spatial features simultaneously, although they rely on local spatial-spectral regions and thus ignore the global spectral similarity and spatial consistency. Meanwhile, some of these methods contain huge model parameters which require a large number of training samples. In this paper, a novel Tensor Singular Spectral Analysis (TensorSSA) method is proposed to extract global and low-rank features of HSI. In TensorSSA, an adaptive embedding operation is first proposed to construct a trajectory tensor corresponding to the entire HSI, which takes full advantage of the spatial similarity and improves the adequate representation of the global low-rank properties of the HSI. Moreover, the obtained trajectory tensor, which contains the global and local spatial and spectral information of the HSI, is decomposed by the Tensor singular value decomposition (t-SVD) to explore its low-rank intrinsic features. Finally, the efficacy of the extracted features is evaluated using the accuracy of image classification with a support vector machine (SVM) classifier. Experimental results on three publicly available datasets have fully demonstrated the superiority of the proposed TensorSSA over a few state-of-the-art 2D/3D feature extraction and deep learning algorithms, even with a limited number of training samples.

Recently, unmixing-based networks have shown significant potential in unsupervised multispectral-aided hyperspectral image super-resolution task (MS-aided HS-SR). Nevertheless, the representation ability of unsupervised networks and the design of loss functions still have not been fully explored, leaving large room for further improvement. To this end, we propose an enhanced unmixing-inspired unsupervised network with attention-embedded degradation learning, EU2ADL for short, to realize MS-aided HS-SR. First, two coupled autoencoders serve as the backbone of EU2ADL network to simultaneously decompose input modalities into abundances and corresponding endmembers, whose encoder part is composed of a spatial-spectral two-stream subnetwork for modality-salient representation learning and a parameter-shared one-stream subnetwork for modality-interacted representation enhancement. More importantly, a hybrid model-constrained loss containing a perceptual abundance term and a degradation-guided term is introduced to further eliminate the latent distortions. Since the hybrid loss is built on the degradation model, we additionally present an attention-embedded degradation learning network to adaptively estimate the unknown degradation parameters. Extensive experimental results on four datasets demonstrate the effectiveness of our proposed methods when compared with state-of-the-arts.

Hyperspectral image super-resolution can compensate for the incompleteness of single-sensor imaging and provide desirable products with both high spatial and spectral resolution. Among them, unmixing-inspired networks have drawn considerable attention owing to their straightforward unsupervised paradigm. However, most do not fully capture and utilize the multi-modal information due to their limited representation ability of constructed networks, hence leaving large room for further improvement. To this end, we propose an X-shaped interactive autoencoders network with cross-modality mutual learning between hyperspectral and multispectral data, XINet for short, to cope with this problem. Generally, it employs a coupled structure equipped with two autoencoders, aiming at deriving latent abundances and corresponding endmembers from input correspondence. Inside the network, a novel X-shaped interactive architecture is designed by coupling two disjointed U-Nets together via a parameter-shared strategy, which not only enables sufficient information flow between two modalities but also leads to informative spatial-spectral features. Considering the complementarity across each modality, a cross-modality mutual learning module is constructed to further transfer knowledge from one modality to another, allowing for better utilization of multi-modal features. Moreover, a joint self-supervised loss is proposed to effectively optimize our proposed XINet, enabling an unsupervised manner without external triplets supervision. Extensive experiments, including super-resolved results in four datasets, robustness analysis, and extension to other applications, are conducted, and the superiority of our method is demonstrated.

Deep learning techniques have become popular in land cover change detection (LCCD) with remote sensing images (RSIs). However, many existing networks mostly concentrate on learning deep features but without considering the effect of different features’ attention and fusion strategy on detection performance. In this paper, a novel hierarchical attention feature fusion (HAFF)-based network for LCCD with RSIs is proposed. In the proposed HAFF-based network, novel multi-scale convolution fusion filters (MCFFs) explore the global semantic feature of the interested targets from multi-perspectives ways. To achieve that objective, the proposed MCFFs are composed by a well-known position attention module (PAM) and a novel multi-perspectives feature filter block with different kernel sizes. In addition, a compound loss function was proposed for balancing the impact from the features at different levels in terms of backpropagation error. Experiments conducted on six pairs of real RSIs, including three pairs of homogeneous images and three pairs of heterogeneous images, confirmed the superiority of the proposed HAFF network over other cognate methods. Moreover, the ablation experiments further confirmed the feasibility and superiority of the proposed MCFFs, whereas quantitative observations indicated that competitive improvements are achieved by the proposed MCFFs in terms of all the evaluation indicators. The code for the proposed approach will be available at https://github.com/ImgSciGroup/HAFF.

Despite of various approaches proposed to smooth the hyperspectral images (HSIs) before feature extraction, the efficacy is still affected by the noise, even using the corrected dataset with the noisy and water absorption bands discarded. In this study, a novel spectral-spatial feature mining framework, Multiscale Superpixelwise Prophet Model (MSPM), is proposed for noise-robust feature extraction and effective classification of the HSI. The prophet model is highly noise-robust for deeply digging into the complex structured features thus enlarging interclass diversity and improving intraclass similarity. First, the superpixelwise segmentation is produced from the first three principal components of an HSI to group pixels into regions with adaptively determined sizes and shapes. A multiscale prophet model is utilized to extract the multiscale informative trend components from the average spectrum of each superpixel. Taking the multiscale trend signal as the input feature, the HSI data are classified superpixelwisely, which is further refined by a majority vote based decision fusion. Comprehensive experiments on three publicly available datasets have fully validated the efficacy and robustness of our MSPM model when benchmarked with eleven state-of-the-art algorithms, including six spectral-spatial methods and five deep learning ones. Besides, MSPM also shows superiority under limited training samples, due to the combined strategies of superpixelwise fusion and multiscale fusion. Our model has provided a useful solution for noise-robust feature extraction as it achieves superior HSI classification even from the uncorrected dataset without prefiltering the water absorption and noisy bands.

Currently, long-range spectral and spatial dependencies have been widely demonstrated to be essential for hyperspectral image (HSI) classification. Due to the transformer superior ability to exploit long-range representations, the transformer-based methods have exhibited enormous potential. However, existing transformer-based approaches still face two crucial issues that hinder the further performance promotion of HSI classification: 1) treating HSI as 1D sequences neglects spatial properties of HSI, 2) the dependence between spectral and spatial information is not fully considered. To tackle the above problems, a large kernel spectral-spatial attention network (LKSSAN) is proposed to capture the long-range 3D properties of HSI, which is inspired by the visual attention network (VAN). Specifically, a spectral-spatial attention module is first proposed to effectively exploit discriminative 3D spectral-spatial features while keeping the 3D structure of HSI. This module introduces the large kernel attention (LKA) and convolution feed-forward (CFF) to flexibly emphasize, model, and exploit the long-range 3D feature dependencies with lower computational pressure. Finally, the features from the spectral-spatial attention module are fed into the classification module for the optimization of 3D spectral-spatial representation. To verify the effectiveness of the proposed classification method, experiments are executed on four widely used HSI data sets. The experiments demonstrate that LKSSAN is indeed an effective way for long-range 3D feature extraction of HSI.

Semantic segmentation of high-resolution aerial images is a challenging task on account of complex scene-variation and large scale-difference. However, these two issues are inadequately addressed in general semantic segmentation methods. In this paper, we propose a Multi-scale Prototype Contrast Network (MPCNet) to improve the adaptive capability for different scenes and scales. Specifically, a novel multi-scale prototype transformer decoder (MPTD) is designed to extract dynamic scene-specific prototypes as pixel classifier by fusing information of feature maps and learnable class tokens. To exploit cross-scene context information and accommodate the large scale-difference in aerial image, we build a multi-scale prototype memory queue to store these multi-scale prototypes during training. Upon the multi-scale prototype memory queue, a novel multi-scale prototype contrastive loss is proposed to increase object feature discriminability across multiple scale, which brings better consistency of intermediate feature and boosts the convergence of network. Extensive experimental results on three publicly available datasets demonstrate the effectiveness and efficiency of our MPCNet over other state-of-the-art methods. The code is available at https://github.com/qixiong-wang/mmsegmentation-mpcnet.

Haze in hyperspectral images (HSIs) can lead to crosstalk between multiple bands, resulting in errors that can be amplified and transmitted during data processing. As a consequence, this may cause a reduction in the accuracy and precision of remote sensing data. The purpose of haze removal is to restore high-quality HSIs from degraded ones. The high spectral resolution and typically dozens to hundreds of spectral bands in HSIs pose significant challenges for haze removal. Thus, many methods designed for natural and multispectral images are not effective in removing haze in HSIs. To address this challenge, we develop a model called asymmetric attention convolution network (AACNet) designed for haze removal in HSIs. Specifically, the basic architecture of AACNet is mainly composed of several residual asymmetric attention groups (RAAGs), where the core components are residual asymmetric attention blocks (RAABs). This design enables the full utilization of deep spatial-spectral features while skipping low-frequency regions and focusing more on the haze-affected areas. To more accurately restore the spectral information in areas polluted by haze, a pooling channel self-attention (PCSA) module has been proposed. This module can effectively reconstruct the spectral response curve that is affected by the haze. Our experiments on both simulated and real datasets demonstrate that the proposed AACNet outperforms several leading haze removal methods in both precision and visual quality. The source code and data of this article will be made publicly available at https://github.com/SZU710/AACNet for reproducible research.

Deep convolutional neural networks (CNNs) have improved remote sensing image analysis, but their high computational demands may limit their deployment on low-end devices with limited resources, such as intelligent satellites and unmanned aerial vehicles. Considering the computation complexity, we propose a Guided Hybrid Quantization with One-to-one Self-Teaching (GHOST) framework. More concretely, we first design a structure called guided quantization self-distillation (GQSD), an innovative idea for realizing a lightweight model through the synergy of quantization and distillation. The training process of the quantization model is guided by its full-precision model, which is time-saving and cost-saving without preparing a huge pre-trained model in advance. Second, we put forward a hybrid quantization (HQ) module that automatically acquires the optimal bit-width by imposing a threshold constraint on the distribution distance between the center point and samples in the weight search space, aiming to retain more shallow detail information that is advantageous for small object detection. Third, to improve information transformation, we propose a one-to-one self-teaching (OST) module to give the student network the ability to self-judgment. A switch control machine (SCM) builds a bridge between the student and teacher networks in the same location to help the teacher reduce wrong guidance and impart vital knowledge about objects without vast background information to the student. This distillation method allows a model to learn from itself and gain substantial improvement without any additional supervision. Extensive experiments on a multimodal dataset (VEDAI) and single-modality datasets (DOTA, NWPU, and DIOR) show that object detection based on GHOST outperforms the existing detectors. The tiny parameters (<9.7 MB) and Bit-Operations (BOPs) (<2158 G) compared with any remote sensing-based, lightweight, or distillation-based algorithms demonstrate the superiority in the lightweight design domain. Our code and model will be released at https://github.com/icey-zhang/GHOST.

Textured 3D mesh is one of the final user products in photogrammetry and remote sensing. However, research on the semantic segmentation of complex urban scenes represented by textured 3D meshes is in its infancy. We present a mesh-based dynamic graph CNN (DGCNN) for the semantic segmentation of textured 3D meshes. To represent each mesh facet, composite input feature vectors are constructed by concatenating the face-inherent features, i.e., XYZ coordinates of the center of gravity (CoG), texture values, and normal vectors. A texture fusion module is embedded into the proposed mesh-based DGCNN to generate high-level semantic features of the high-resolution texture information, which is useful for semantic segmentation. We achieve competitive accuracies when the proposed method is applied to the SUM mesh datasets. The overall accuracy (OA), Kappa coefficient (Kap), mean precision (mP), mean recall (mR), mean F1 score (mF1), and mean intersection over union (mIoU) are 93.3%, 88.7%, 79.6%, 83.0%, 80.7%, and 69.6%, respectively. In particular, the OA, mean class accuracy (mAcc), mIoU, and mF1 increase by 0.3%, 12.4%, 3.4%, and 6.9%, respectively, compared to the state-of-the-art method.

Extracting buildings from remote sensing images plays an important role in urban development planning, disaster assessment and mapping. Convolutional neural network (CNN) has been widely applied to building extraction because of its powerful deep semantic feature extraction ability. However, existing CNN-based building extraction methods are difficult to accurately extract multiscale buildings with accurate edges because of the limitation of feature receptive fields and the loss of spatial detail information. For the above problems, this paper proposes a multiscale receptive field network (MSRF-Net) to accurately extract multiscale buildings from remote sensing images. MSRF-Net includes multiscale receptive field feature encoder (MRFF-Encoder) and multipath decoder. In the MRFF-Encoder, a multiscale attentional down (MSAD) module and asymmetric residual inception (ARI) module are proposed to capture multiscale receptive field features. In the multipath decoder, convolutions with different kernel size and dilation are used in three parallel paths to learn localization-preserved multiscale features with multiscale receptive field. What’s more, the features of different branches and MRFF-Encoder are fused by the proposed feature combination module, which contribute to capture context information of multiscale receptive field while recovering the resolution of feature space. The experimental results show that compared with the latest MAP-Net, MSRF-Net has achieved F1 score growth of 1.14%, 0.42%, 1.11% and IoU score growth of 1.68%, 0.76% and 1.64% respectively on Massachusetts data set, WHU data set and the Typical Cities Building data set.

Well-known deep learning (DL) is widely used in fusion based hyperspectral image super-resolution (HS-SR). However, DL-based HS-SR models have been designed mostly using off-the-shelf components from current deep learning toolkits, which lead to two inherent challenges: i) they have largely ignored the prior information contained in the observed images, which may cause the output of the network to deviate from the general prior configuration; ii) they are not specifically designed for HS-SR, making it hard to intuitively understand its implementation mechanism and therefore uninterpretable. In this paper, we propose a noise prior knowledge informed Bayesian inference network for HS-SR. Instead of designing a "black-box" deep model, our proposed network, termed as BayeSR, reasonably embeds the Bayesian inference with the Gaussian noise prior assumption to the deep neural network. In particular, we first construct a Bayesian inference model with the Gaussian noise prior assumption that can be solved iteratively by the proximal gradient algorithm, and then convert each operator involved in the iterative algorithm into a specific form of network connection to construct an unfolding network. In the process of network unfolding, based on the characteristics of the noise matrix, we ingeniously convert the diagonal noise matrix operation which represents the noise variance of each band into the channel attention. As a result, the proposed BayeSR explicitly encodes the prior knowledge possessed by the observed images and considers the intrinsic generation mechanism of HS-SR through the whole network flow. Qualitative and quantitative experimental results demonstrate the superiority of the proposed BayeSR against some state-of-the-art methods.

The synthesis of high-resolution (HR) hyperspectral image (HSI) by fusing a low-resolution HSI with a corresponding HR multispectral image has emerged as a prevalent HSI super-resolution (HSR) scheme. Recent researches have revealed that tensor analysis is an emerging tool for HSR. However, most off-the-shelf tensor-based HSR algorithms tend to encounter challenges in rank determination and modeling capacity. To address these issues, we construct nonlocal patch tensors (NPTs) and characterize low-rank structures with coupled Bayesian tensor factorization. It is worth emphasizing that the intrinsic global spectral correlation and nonlocal spatial similarity can be simultaneously explored under the proposed model. Moreover, benefiting from the technique of automatic relevance determination, we propose a hierarchical probabilistic framework based on Canonical Polyadic (CP) factorization, which incorporates a sparsity-inducing prior over the underlying factor matrices. We further develop an effective expectation-maximization-type optimization scheme for framework estimation. In contrast to existing works, the proposed model can infer the latent CP rank of NPT adaptively without tuning parameters. Extensive experiments on synthesized and real datasets illustrate the intrinsic capability of our model in rank determination as well as its superiority in fusion performance.

Deep learning (DL) based methods represented by convolutional neural networks (CNNs) are widely used in hyperspectral image classification (HSIC). Some of these methods have strong ability to extract local information, but the extraction of long-range features is slightly inefficient, while others are just the opposite. For example, limited by the receptive fields, CNN is difficult to capture the contextual spectral-spatial features from a long-range spectral-spatial relationship. Besides, the success of DL-based methods is greatly attributed to numerous labeled samples, whose acquisition are time-consuming and cost-consuming. To resolve these problems, a hyperspectral classification framework based on multi-attention Transformer (MAT) and adaptive superpixel segmentation-based active learning (MAT-ASSAL) is proposed, which successfully achieves excellent classification performance, especially under the condition of small-size samples. Firstly, a multi-attention Transformer network is built for HSIC. Specifically, the self-attention module of Transformer is applied to model long-range contextual dependency between spectral-spatial embedding. Moreover, in order to capture local features, an outlook-attention module which can efficiently encode fine-level features and contexts into tokens is utilized to improve the correlation between the center spectral-spatial embedding and its surroundings. Secondly, aiming to train a excellent MAT model through limited labeled samples, a novel active learning (AL) based on superpixel segmentation is proposed to select important samples for MAT. Finally, to better integrate local spatial similarity into active learning, an adaptive superpixel (SP) segmentation algorithm, which can save SPs in uninformative regions and preserve edge details in complex regions, is employed to generate better local spatial constraints for AL. Quantitative and qualitative results indicate that the MAT-ASSAL outperforms seven state-of-the-art methods on three HSI datasets.

Weakly supervised solar panel mapping has shown its advantages in automatically detecting solar panels from remote sensing images with low annotation costs. Considering the noisy nature of pseudo labels (PLs), which are frequently employed in weakly supervised methods, we propose to introduce uncertainty measure to guide the estimation of noise levels in PLs and develop a novel method based on uncertainty-aware label transition (UALT). The proposed method consists of three parts: uncertainty estimation network, uncertainty-aware label transition network, and target mapping network with forward correction. We first generate heteroscedastic uncertainty by learning an estimator under Bayes formalism. Then, with the uncertainty as guidance, a label transition network is trained to learn the mapping between clean labels, and Bayes optimal labels and predict the instance-dependent transition matrix. Finally, the transition matrix is employed in the forward correction process, where the target mapping network produces clean predictions for solar panels. Comparative experiments with six state-of-the-art weakly supervised methods on an aerial image data set show the superiority of the proposed UALT, especially in mapping accuracy and dis-covering small-scale objects.

Solar panel mapping from high-resolution aerial images is becoming increasingly crucial to grid planning and operation, where weakly supervised approach has been explored. To cope with the noisy nature of pseudo-labels (PLs) generated by weakly supervised object localization, we propose an effective uncertainty-aware forward correction (UA-FC) method to learn clean predictions from the noisy PLs. The proposed method consists of two steps: heteroscedastic uncertainty estimation and forward correction procedure. The purpose of the first step is to produce uncertainty as an indicator for the instance-dependent noise. The second step includes a target mapping network to produce clean predictions and a transition function to model the relationship between clean predictions and noisy PLs. As estimating every probability of one class flipped into another is difficult and time-consuming, we introduce heteroscedastic uncertainty as a measurement and propose an uncertainty-based transformation function to map clean predictions into noisy ones. By minimizing the errors between the noisy predictions and noisy labels, the target mapping network is able to offer clean predictions close to the actual objects. Extensive experiments on an aerial dataset reveal that the proposed method outperforms other state-of-the-art methods by a large margin, especially in recovering the boundary of the objects.

With satellite platforms gazing at a target territory, the captured satellite videos exhibit local misalignment and local intensity variation on some stationary objects that can be mistakenly extracted as moving objects and increase false alarm rates. Typical approaches for mitigating the effect of moving cameras in moving object detection (MOD) follow domain transformation technique, where the misalignment between consecutive frames is restricted to the image planar. However, such technique cannot properly handle satellite videos, as the local misalignment on them is caused by the varying projections from the 3D objects on the Earth's surface to 2D image planar. In order to suppress the effect of moving satellite platform in MOD, we propose a Moving-Confidence-Assisted Matrix Decomposition (MCMD) model, where foreground regularization is designed to promote real moving objects and ignore system movements with the assistance of a moving-confidence score estimated from dense optical flows. For solving the convex optimization problem in MCMD, both batch processing and online solutions are developed in this study, by adopting the alternating direction method and the stochastic optimization strategy, respectively. Experimental results on the videos captured by SkySat and Jilin-1 show that MCMD outperforms the state-of-the-art techniques with improved precision by suppressing effect of nonstationary satellite platforms.

Singular spectral analysis (SSA) has recently been successfully applied to feature extraction in hyperspectral image (HSI), including conventional (1-D) SSA in spectral domain and 2-D SSA in spatial domain. However, there are some drawbacks, such as sensitivity to the window size, high computational complexity under a large window, and failing to extract joint spectral-spatial features. To tackle these issues, in this article, we propose superpixelwise adaptive SSA (SpaSSA), that is superpixelwise adaptive SSA for exploiting local spatial information of HSI. The extraction of local (instead of global) features, particularly in HSI, can be more effective for characterizing the objects within an image. In SpaSSA, conventional SSA and 2-D SSA are combined and adaptively applied to each superpixel derived from an oversegmented HSI. According to the size of the derived superpixels, either SSA or 2-D singular spectrum analysis (2D-SSA) is adaptively applied for feature extraction, where the embedding window in 2D-SSA is also adaptive to the size of the superpixel. Experimental results on the three datasets have shown that the proposed SpaSSA outperforms both SSA and 2D-SSA in terms of classification accuracy and computational complexity. By combining SpaSSA with the principal component analysis (SpaSSA-PCA), the accuracy of land-cover analysis can be further improved, outperforming several state-of-the-art approaches.

In remote sensing scene classification (RSSC), features can be extracted with different spatial frequencies where high-frequency features usually represent detailed information and low-frequency features usually represent global structures. However, it is challenging to extract meaningful semantic information for RSSC tasks by just utilizing high- or low-frequency features. The spatial composition of remote sensing images (RSIs) is more complex than that of natural images, and the scales of objects vary significantly. In this article, a multiscale feature fusion covariance network (MF2CNet) with octave convolution (Oct Conv) is proposed, which can extract multifrequency and multiscale features from RSIs. First, the multifrequency feature extraction (MFE) module is used to obtain fine-grained frequency features by Oct Conv. Then, the features of different layers in MF2CNet are fused by the multiscale feature fusion (MF2) module. Finally, instead of using global average pooling (GAP), global covariance pooling (GCP) extracts high-order information from RSIs to capture richer statistics of deep features. In the proposed MF2CNet, the obtained multifrequency and multiscale features can effectively improve the performance of CNNs. Experimental results on four public RSI datasets show that MF2CNet has advantages in RSSC over current state-of-the-art methods. The source codes of this method can be found athttps://github.com/liuqingxin-chd/MF2CNet.

Automatic weed monitoring and classification are critical for effective site-specific weed management. With the increasing availability of different sensors, it is possible for weed management to be achieved by processing a wide range of images captured from various remote sensing platforms. A deep learning-based convolutional neural network (CNN) can learn the sophisticated spectral, spatial, and structural features to discriminate weed species. The challenge is to train a CNN architecture for each dataset with limited training samples. In this study, we develop a partial transferable CNN to cope with a new dataset with a different spatial resolution, a different number of bands, and variation in the signal-to-noise ratio. The goal is to make the training for each new dataset less demanding. We conducted a series of experiments on simulated image datasets from two sensors. This study reveals that the dropout layers between the convolutional layers have a significant impact for partial transferable CNN. Even-numbered subset layers from source CNN has a stronger impact on dealing with a task of different spatial resolution. For a different number of bands in source and target datasets, except for the first convolutional layer, the remaining layers are used for the analysis. Results show that network transfer is possible when the numbers of bands of the two datasets are not very different. For the variation in signal-to-noise ratio, it is found that the performance of transfer learning is acceptable when the noise level is not high. Based on these findings, experiments were conducted on two real datasets from two sensors, which includes all the variations. The comparison results using different state-of-the-art models show that partial CNN transfer with even-numbered layers provides better mapping accuracy for the target dataset with a limited number of training samples.

As hyperspectral imagery (HSI) contains rich spectral and spatial information, a novel principal component analysis (PCA) and segmented-PCA (SPCA)-based multiscale 2-D-singular spectrum analysis (2-D-SSA) fusion method is proposed for joint spectral–spatial HSI feature extraction and classification. Considering the overall spectra and adjacent band correlations of objects, the PCA and SPCA methods are utilized first for spectral dimension reduction, respectively. Then, multiscale 2-D-SSA is applied onto the SPCA dimension-reduced images to extract abundant spatial features at different scales, where PCA is applied again for dimensionality reduction. The obtained multiscale spatial features are then fused with the global spectral features derived from PCA to form multiscale spectral–spatial features (MSF-PCs). The performance of the extracted MSF-PCs is evaluated using the support vector machine (SVM) classifier. Experiments on four benchmark HSI data sets have shown that the proposed method outperforms other state-of-the-art feature extraction methods, including several deep learning approaches, when only a small number of training samples are available.

As an essential reprocessing method, dimensionality reduction (DR) can reduce the data redundancy and improve the performance of hyperspectral image (HSI) classification. A novel unsupervised DR framework with feature interpretability, which integrates both band selection (BS) and spatial noise reduction method, is proposed to extract low-dimensional spectral-spatial features of HSI. We proposed a new Neighboring band Grouping and Normalized Matching Filter (NGNMF) for BS, which can reduce the data dimension whilst preserve the corresponding spectral information. An enhanced 2-D singular spectrum analysis (E2DSSA) method is also proposed to extract the spatial context and structural information from each selected band, aiming to decrease the intra-class variability and reduce the effect of noise in the spatial domain. The support vector machine (SVM) classifier is used to evaluate the effectiveness of the extracted spectral-spatial low-dimensional features. Experimental results on three publicly available HSI datasets have fully demonstrated the efficacy of the proposed NGNMF-E2DSSA method, which has surpassed a number of state-of-the-art DR methods.

With the development of hyperspectral imaging technology, hyperspectral images (HSIs) have become important when analyzing the class of ground objects. In recent years, benefiting from the massive labeled data, deep learning has achieved a series of breakthroughs in many fields of research. However, labeling HSIs requires sufficient domain knowledge and is time-consuming and laborious. Thus, how to apply deep learning effectively to small labeled samples is an important topic of research in HSI classification. To solve this problem, we propose a semisupervised Siamese network that embeds Siamese network into a semisupervised learning scheme. It integrates an autoencoder module and a Siamese network to, respectively, investigate information in a large amount of unlabeled data and rectify it with a limited labeled sample set, which is called 3DAES. First, the autoencoder method is trained on the massive unlabeled data to learn the refinement representation, creating an unsupervised feature. Second, based on this unsupervised feature, limited labeled samples are used to train a Siamese network to rectify the unsupervised feature to improve feature separability among various classes. Furthermore, by training the Siamese network, a random sampling scheme is used to accelerate training and avoid imbalance among various sample classes. Experiments on three benchmark HSI datasets consistently demonstrate the effectiveness and robustness of the proposed 3DAES approach with limited labeled samples. For study replication, the code developed for this study is available athttps://github.com/ShuGuoJ/3DAES.git.

Due to the detailed spectral information through hundreds of narrow spectral bands provided by hyperspectral image (HSI) data, it can be employed to accurately classify diverse materials of interest, which is one of the core applications of hyperspectral remote sensing technology. In recent years, with the rapid development of deep learning, convolutional neural networks (CNNs) have been successfully applied in many fields, including HSI classification. However, the random gradient descent-based parameter updating scheme is too general and leading to the inefficiency of CNN models. Moreover, the high dimensionality and limited training samples of HSI data also exacerbate the overfitting problem. To tackle these issues, in this article, a novel deep network with multilayer and multibranch architecture, named 3-D Gabor CNN (3DG-CNN), is proposed for HSI classification. More precisely, since the predefined 3-D Gabor filters in multiple scales and orientations could well characterize the internal spatial–spectral structure of HSI data from various perspectives, the 3-D Gabor-modulated kernels (3-D GMKs) are employed to replace the random initialization kernels. Moreover, the specially designed multibranch architecture enables the network to better integrating the scalable property of 3-D Gabor filters; thus, the representative ability and robustness of the extracted features can be greatly improved. Alternatively, the number of network parameters is substantially reduced due to the incorporation of 3-D Gabor modulation, relieving the training complexity and also alleviating the training process from overfitting. Experimental results on four real HSI datasets (including two newly released ones in the literature) have demonstrated that the proposed 3DG-CNN model can achieve better performance than several widely used machine-learning-based and deep-learning-based approaches. For the sake of reproducibility, the codes of the proposed 3DG-CNN model are available athttp://jiasen.tech/papers/.

Domain adaptation, which cleverly applies the classifier learned from the source domain with sufficient labeled samples to the target domain with limited labeled samples, provides a feasible alternative to handle the small training sample problem of hyperspectral image (HSI) classification and has attracted much attention in the research field recently. Apparently, feature discriminative ability is vital for domain adaptation, which plays a crucial role during the migration process of transfer learning. In this article, a gradient feature-oriented 3-D domain adaptation (GF-3DDA) approach is proposed for HSI classification. First, 3-D Gabor is employed to remove noise from the original data, and two 2-D gradient-based features, 2-D Sobel gradient (SG) and 2-D derivative-of-Gaussian (DtG), are extended to the 3-D domain to coincide with the integrated spatial–spectral organization of HSI. Thus, the 3-D Sobel–Gabor gradient (3DSGG) and 3-D derivative-of-Gaussian-Gabor (3DDGG) features are achieved. Second, a 3-D domain adaptation method is implemented to jointly exploit the second- and fourth-order statistical descriptors in the spatial–spectral dimensions, which could effectively reduce domain shifts and thus achieve improved domain adaptation. Third, all the extracted domain-adapted feature modules are collaboratively classified by extreme learning machine (ELM), and the probability-like outputs of every ELM classifier are combined together to accomplish the classification task. Four hyperspectral data sets that each contains two scenes, i.e., Pavia, Shanghai–Hangzhou, Indiana, and Houston, are tested in the experiments. When only ten labeled samples per class are used in the target domain, the classification accuracies on four hyperspectral data sets achieved by our GF-3DDA approach are 93.31%, 84.35%, 69.32%, and 80.06%, respectively.

Image super-resolution (SR) methods can generate remote sensing images with high spatial resolution without increasing the cost of acquisition equipment, thereby providing a feasible way to improve the quality of remote sensing images. Clearly, image SR is a severe ill-posed problem. With the development of deep learning, the powerful fitting ability of deep neural networks has solved this problem to some extent. Since the texture information of various remote sensing images are totally different from each other, in this paper, we proposed a network based on generative adversarial network (GAN) to achieve high resolution remote sensing images, named multi-attention generative adversarial network (MA-GAN). The main body of the generator in MA-GAN contains three blocks: pyramid-convolutional residualdense (PCRD) block, attention-based upsampling (AUP) block and attention-based fusion (AF) block. Specifically, the developed attention-pyramid convolutional (AttPConv) operator in PCRD block combines multi-scale convolution and channel attention (CA) to automatically learn and adjust the scale of residuals for better representation. The established AUP block utilizes pixel attention (PA) to perform arbitrary scales of upsampling. And the AF block employs branch attention (BA) to integrate upsampled low-resolution images with high-level features. Besides, the loss function takes both adversarial loss and feature loss into consideration to guide the learning procedure of generator. We have compared our MA-GAN approach with several state-of-the-art methods on a number of remote sensing scenes, and experimental results consistently demonstrate the effectiveness of the proposed MA-GAN. For study replication, the source code will be released at: https://github.com/ZhihaoWang1997/MA-GAN.

Hyperspectral imagery (HSI) contains hundreds of bands, which provide a wealth of spectral information and enable better characterization of features. However, the excessive dimensionality also poses a dimensional disaster for subsequent processing. Fortunately, band selection (BS) gives a straightforward and effective way to pick out a subset of bands with rich information and low correlation. Although many hyperspectral BS methods, especially clustering-based ones, have been proposed by researchers in recent years, the contextual information of adjacent bands and the spatial structural information of materials are not well investigated. Therefore, in this article, a multiscale superpixel-level group-clustering framework (MSGCF) has been proposed for hyperspectral BS. Different from previous, a new superpixel-level distance measure is elaborately utilized to group and cluster the spectral bands, which jointly considers the spectral context and spatial structure information. Concretely, to preserve the spatial structural information of HSI, multiple superpixel segmentation is first performed to generate superpixel maps in multiscales, which enables complementarity of multiple superpixel segmentation algorithms and adaptation to diverse scales of land cover types. Second, the grouping and clustering paradigm is introduced to conduct the contextual information among bands. Here the maximum points of superpixel-level KL-$\ell _{1}$distance of adjacent bands are adopted as partition points to separate bands into groups, which encourages adjacent bands with strong correlation to be divided into the same group. Third, a superpixel-level fast density-based clustering method (SuFDPC) with superpixel-level$\ell _{2, 1}$distance is developed to select representative bands in every group. Finally, BS results are achieved with a ranking-based voting strategy by concerning information entropy and frequency of occurrence in a unified scheme. A series of ablation analyses and experimental comparisons on four real HSI datasets have been conducted, as well as similarity comparisons for the selected bands. The experimental results consistently demonstrated the effectiveness of our MSGCF approach. The codes of this work will be available athttp://jiasen.tech/papers/for the sake of reproducibility.

Large-scene precise classification of multispectral images (MSIs) has become one of the hot topics in remote sensing field. MSIs usually have wide swath and a meter or even submeter level of spatial resolution, which make large-scene observation possible. However, the limited number of spectral bands leads to the confusion of land covers in classification, especially for the large-scene conditions with abundant land cover types. Therefore, overlapped hyperspectral images (HSIs) can be used to improve the precision degree of classification. To achieve this purpose, coupled dictionary learning has been proposed as a major means. Aiming at separating the class-specific characteristics and mutual patterns among different land covers, this paper proposed a separable coupled dictionary learning (SCDL) method, which converts the separation of mutual features into the construction of separable coupled dictionaries and learns both class-specific coupled dictionaries and mutual coupled dictionaries simultaneously with the aid of label information. More specifically, the proposed method uses the labels of training samples to construct class-specific reconstruction error constraint, class-specificity constraint and separable dictionary incoherence constraint as regularization terms, to make sure that the learned coupled dictionaries to be both compact and discriminative. The learned separable coupled dictionaries facilitate pixels belong to the same category to be represented by the mutual dictionary and the class-specific sub-dictionary of corresponding class. The experiments compared with several state-of-the-art methods on three pairs of HSI and MSI have shown better classification performance.

Deep learning methods have demonstrated excellent performance in hyperspectral image (HSI) classification. However, these methods mainly focus on improving the classification accuracy while ignoring their high complexity. By considering that the data formats of both HSIs and network weights can be represented in the form of tensors, we develop a new lightweight tensorized neural network for HSI classification that takes advantage of low-rank tensor decomposition techniques to reduce complexity. Firstly, inspired by tensor train (TT)-based tensorized convolutional layers, a new tensorized 2D convolutional layer based on chain calculation (with better expression ability) is introduced. Based on this innovation, a new lightweight 2D tensorized neural network (2D-TNN) is designed for HSI classification. Furthermore, to better preserve the intrinsic structure of HSI data, a new lightweight 3D tensorized neural network (3D-TNN) is proposed by extending the tensorized 2D convolutional layers to their 3D versions. Quantitative and comparative experiments on three widely used data sets show that the proposed models are able to achieve state-of-the-art performance (with a low number of model parameters) for different training sample sizes, especially for very small training sets.

With the rapid development of the solar distribution, solar panel mapping is becoming increasingly valuable to decision-makers. Weakly supervised methods have been developed to reduce the cost in training sample collection, and the most successful ones follow the alternative training scheme, which first generates coarse object localizations as pseudo labels (PLs) and then utilizes these PLs to train an end-to-end network for object extraction. As remote sensing images are typically characterized by multiple occurrences of objects and complicated backgrounds, the alternative training scheme suffers from low mapping accuracy and deficient boundary maintenance due to the varying quality of PLs. In this article, we focus on addressing these problems by adaptively adjusting the contributions of quality-varying PLs and propose a novel self-paced residual aggregated network (SP-RAN) for solar panel mapping. Specifically, with the initial PLs generated by gradient-weighted class activation mapping, a residual aggregated network is designed for target mapping with special consideration for the capability in producing complete and well-shaped mapping results. Considering the inconsistent quality of PLs, an effective confidence-aware (CA) loss is developed to emphasize the contribution of high-quality PLs and alleviate the negative impacts brought by the bad-quality ones in the training phase. Moreover, to concentrate on boundary maintenance, a novel self-paced label correction (SP-LC) strategy is proposed to selectively update PLs by considering their reliability. Extensive experimental comparisons with state-of-the-art methods and ablation study on two aerial datasets and a remote sensing dataset demonstrate the superiority of the proposed method.

This study presents a spectral–spatial self-attention network (SSSAN) for classification of hyperspectral images (HSIs), which can adaptively integrate local features with long-range dependencies related to the pixel to be classified. Specifically, it has two subnetworks. The spatial subnetwork introduces the proposed spatial self-attention module to exploit rich patch-based contextual information related to the center pixel. The spectral subnetwork introduces the proposed spectral self-attention module to exploit the long-range spectral correlation over local spectral features. The extracted spectral and spatial features are then adaptively fused for HSI classification. Experiments conducted on four HSI datasets demonstrate that the proposed network outperforms several state-of-the-art methods.

Integration of spectral and spatial information is extremely important for the classification of high-resolution hyperspectral images (HSIs). Gravitation describes interaction among celestial bodies which can be applied to measure similarity between data for image classification. However, gravitation is hard to combine with spatial information and rarely been applied in HSI classification. This paper proposes a Bayesian Gravitation based Classification (BGC) to integrate the spectral and spatial information of local neighbors and training samples. In the BGC method, each testing pixel is first assumed as a massive object with unit volume and a particular density, where the density is taken as the data mass in BGC. Specifically, the data mass is formulated as an exponential function of the spectral distribution of its neighbors and the spatial prior distribution of its surrounding training samples based on the Bayesian theorem. Then, a joint data gravitation model is developed as the classification measure, in which the data mass is taken to weigh the contribution of different neighbors in a local region. Four benchmark HSI datasets, i.e. the Indian Pines, Pavia University, Salinas, and Grss_dfc_2014, are tested to verify the BGC method. The experimental results are compared with that of several well-known HSI classification methods, including the support vector machines, sparse representation, and other eight state-of-the-art HSI classification methods. The BGC shows apparent superiority in the classification of high-resolution HSIs and also flexibility for HSIs with limited samples.

3-D Gabor, as a typical filter, plays a critical role in extracting discriminative spectral–spatial features from hyperspectral images (HSIs). However, the performance of traditional 3-D Gabor is limited by the uniform response to each direction, which is inconsistent with the complexity of land cover distribution. It has been a continuing concern for researchers to investigate the anisotropic 3-D Gabor filters. In addition, the 3-D Gabor wavelets do not make full use of spatial distribution information, thus reducing the accuracy. This article proposes a superpixel-guided variable 3-D Gabor phase coding fusion (SuVGF) framework for HSI classification with limited training samples. First, the variable 3-D Gabor filters are created based on various asymmetric sinusoidal waves and spatial kernel sizes to achieve multidirectional features. Second, the local Gabor phase ternary pattern is adopted to encode the Gabor phases and improve the feature discrimination. Meanwhile, a scale map is produced by the majority voting of multiscale simple noniterative clustering (SNIC) and entropy rate superpixel (ERS) segmentation, which contains sufficient and complementary spatial distribution information. Then, geometric optimization is employed on the scale map to reduce noise disturbances. Finally, all Gabor features are modified by the filter with the guidance of a scale map and fused together as a confidence cube, and the random forest algorithm is exploited for classification. TheSuVGF is applied to three real hyperspectral datasets to demonstrate the superiority of higher accuracy, stronger robustness, and less computational complexity in comparison with several state-of-the-art ones.

In recent years, several representation models based on total variation (TV) have been proposed for hyperspectral imagery (HSI) anomaly detection. However, the TV terms of these works are directly imposed on the representation coefficient matrix, which can destroy the spatial structure of an HSI to some extent. Besides, as the spatial resolution of an HSI is relatively low, mixed pixels existing in an HSI can lead to anomaly component contamination, which can make the difference between background and anomalies not significant enough. To address these issues, a novel enhanced TV (ETV) with an endmember background dictionary (EBD) for hyperspectral anomaly detection is proposed. The ETV is designed to be used on the row vectors of the representation coefficient matrix to enhance the spatial structure of an HSI in the presentation process. Furthermore, the proposed ETV regularized representation model with EBD (ETVEBD) method elaborates on a background dictionary constructed by endmembers of background pixels, which are pure spectral signatures of background pixels. The proposed EBD can decrease the influence of anomaly components in mixed pixels, and the coefficient matrix of the EBD has more physical meanings. The proposed method is evaluated on four hyperspectral datasets, and the experiment results show that its performance is the best compared with the other seven state-of-the-art methods.

Deformable convolutional networks (DCNs) can mitigate the inherent limited geometric transformation. We reformulate the spatialwise attention mechanism using DCNs in this article for semantic segmentation of high-resolution remote sensing (HRRS) images. It combines the sparse spatial sampling strategy and the long-range relationship modeling capability, namely, deformable attention module (DAM). Such locality awareness, more adaptable to HRRS image structures, can capture each pixel’s neighboring structural information. A reasonable multiscale deformable attention net (MDANet) is designed for the HRRS image semantic segmentation with a slightly increased computational cost based on the proposed DAM. Specifically, standard convolutional layers in the raw ResNet50 are equipped with a DAM to control sampling over a broader range of feature levels and aggregate multiscale context information. The experimental results evaluated on Vaihingen and DeepGlobe Land Cover Classification datasets show that the performance accuracy of MDANet is improved by 7.77% and 8.45% compared with the backbone network (ResNet50) in terms of Miou evaluation, respectively. Furthermore, a DAM can perform better than a global spatial attention mechanism with less computation on the$3 \times 64 \times 64$feature map. In addition, the added ablation studies demonstrate the effectiveness and efficiency of the DAM and multiscale strategy, respectively. Moreover, the sensitivity of critical hyperparameters is analyzed.

Hyperspectral image produces high spectral resolution at the sacrifice of spatial resolution. Without reducing the spectral resolution, improving the resolution in the spatial domain is a very challenging problem. Motivated by the discovery that hyperspectral image exhibits high similarity between adjacent bands in a large spectral range, in this paper, we explore a new structure for hyperspectral image super-resolution (DualSR), leading to a dual-stage design, i.e., coarse stage and fine stage. In coarse stage, five bands with high similarity in a certain spectral range are divided into three groups, and the current band is guided to study the potential knowledge. Under the action of alternative spectral fusion mechanism, the coarse SR image is super-resolved in band-by-band. In order to build model from a global perspective, an enhanced back-projection method via spectral angle constraint is developed in fine stage to learn the content of spatial-spectral consistency, dramatically improving the performance gain. Extensive experiments demonstrate the effectiveness of the proposed coarse stage and fine stage. Besides, our network produces state-of-the-art results against existing works in terms of spatial reconstruction and spectral fidelity. Our code is publicly available at https://github.com/qianngli/DualSR.

With the rapid development of solar distribution, mapping the locations and sizes of solar panels is becoming increasingly valuable. Weakly supervised methods have been proposed to reduce the reliance on expensive pixel-wise annotations by adopting weak labels, but inevitably suffer from low mapping accuracy and poor boundary maintenance. To solve these problems, a novel weakly supervised residual aggregated network (WS-RAN) is presented in this paper. In the WS-RAN, pixel-wise labels are automatically generated from image-level labels by a classification network. Then, the produced pixel-wise labels are used to train the residual aggregated network (RAN) for target mapping, which is designed to cope the variations in size and shapes of individual solar panel layout. Particularly, for better boundary maintenance, the residual aggregated block is developed as the basic module in the bottom-up path of the RAN. Experiment results reveal that the proposed WS-RAN significantly outperforms four state-of-the-art weakly supervised methods with 76.0% F1score and 61.3% IoU score.

Moving Object Detection (MOD) from satellite videos plays one of the most fundamental roles in satellite video surveillance. Training a supervised moving object detector typically requires boundary box annotations for object instances, however, this annotation process is time-consuming for satellite videos. In this paper, we propose a weakly supervised method for sidestepping this process, where the supervised information for detecting moving objects on a frame is instead provided by the unsupervised method based on motion knowledge across a video. We adopt the Extended Low-rank and Structured Sparse Decomposition (E-LSD) approach for generating pixel-wise pseudo labels for moving objects. Then the extracted pseudo labels are used for training a Deep Convolutional Neural Network (DCNN) following the Encoder-Decoder architecture with lateral connections for segmenting moving objects from a new frame. We demonstrate the effectiveness of the proposed method on a satellite video dataset, and, compared with five state-of-the-art MOD methods tested, it achieves both improved detection accuracy and promising frame rate.

Very high-resolution (VHR) remote sensing images can provide fine but sometimes trivial ground object details; thus, the semantic labeling of VHR images is a challenging task. To improve the VHR labeling performance, spatial multiscale information and channel attention have been employed recently. However, the exploitation of global object features is still limited, which leads to the loss of capturing within-class variation from location to location. In this letter, we present a multibranch spatial-channel attention (MSCA) model to efficiently extract global dependency and combine it with multiscale and channel attention methods. In the spatial multiscale attention block, a multibranch feature fusion model is established to exploit the global relationship captured by self-attention and the multiscale correlation learned from dilated convolutions. To alleviate the computational cost of pixel-by-pixel self-attention operation, a spatial pyramid compressing method is also designed. In the channel attention block, average and max global pooling strategies are applied, respectively, in two channel attention branches to generalize global information from different perspectives. Those two blocks are then adaptively united by learnable weighting parameters. Experiments on two VHR image data sets demonstrate that the proposed network can yield better performance in comparison with state-of-the-art labeling methods tested.

Hyperspectral images (HSIs) contain rich context information in the spatial domain and spectral domain. To fully explore that information, a data-driven joint spatial–spectral attention network (JSSAN) is proposed in this letter. Specifically, we first design a spatial–spectral attention ($\text{S}^{2}\text{A}$) block to simultaneously capture long-range interdependency of spatial and spectral data via the similarity evaluation. Then we adopt a weighted sum operation of features at all spatial positions and channels to selectively aggregate discriminative spatial–spectral features. Second, the$\text{S}^{2}\text{A}$block is inserted into simple convolutional neural network (CNN) structure to extract more representative features for classification, by adaptively emphasizing features of informative land covers and spectral bands which contribute more to class identification. The experimental results reveal that our proposed method outperforms several state-of-the-art algorithms.

In this letter, we proposed two novel feature selection methods using sparse representation for one-class classification of remote sensing images. In the first method, a sparse reconstructive weight matrix of the data set was obtained by reconstructing samples using sparse representation. The “good” features were then selected by evaluating reconstructing errors in weight matrix. The method is called feature selection based on sample reconstruction (FSSR). In the second method, the weight matrix was obtained by reconstructing features using sparse representation. Feature selection was then conducted by evaluating correlation among features using weight matrix. The method is called feature selection based on feature reconstruction (FSFR). The proposed feature selection methods were evaluated and compared with several state-of-the-art feature selection methods in two different case studies. The experimental results indicate that the proposed methods generally outperformed the existing methods. In particular, FSFR produced stable better performance.

Super-resolution (SR) techniques play a crucial role in increasing the spatial resolution of remote sensing data and overcoming the physical limitations of the spaceborne imaging systems. Though the convolutional neural network (CNN)-based methods have obtained good performance, they show limited capacity when coping with large-scale super-resolving tasks. The more complicated spatial distribution of remote sensing data further increases the difficulty in reconstruction. This article develops a dense-sampling super-resolution network (DSSR) to explore the large-scale SR reconstruction of the remote sensing imageries. Specifically, a dense-sampling mechanism, which reuses an upscaler to upsample multiple low-dimension features, is presented to make the network jointly consider multilevel priors when performing reconstruction. A wide feature attention block (WAB), which incorporates the wide activation and attention mechanism, is introduced to enhance the representation ability of the network. In addition, a chain training strategy is proposed to optimize further the performance of the large-scale models by borrowing knowledge from the pretrained small-scale models. Extensive experiments demonstrate the effectiveness of the proposed methods and show that the DSSR outperforms the state-of-the-art models in both quantitative evaluation and visual quality.

Remotely sensed images, especially in urban areas, have highly complex spatial distribution, since the ground objects have diverse ranges of sizes and shapes. This largely increases the difficulty of super-resolution (SR) tasks. Current deep convolutional neural network (CNN)-based SR methods often show limited performance when coping with complicated images. This article develops a second-order multi-scale super-resolution network (SMSR) to explore reconstruction tasks for difficult cases. Specifically, we propose a single-path feature reuse which cleverly captures multi-scale feature information through aggregating the features learned at different depths of a single path. Further, we present a second-order learning mechanism, which double reuses small-difference and large-difference features at local and global levels, makes use of the learned multi-scale information at maximum. The proposed methods achieve multi-scale learning using small-size convolution only, resulting in a lightweight and high-performance SR network. Experimental results show the superiority of our SMSR over state-of-the-art methods in super-resolving complicated image patterns. The effectiveness of SMSR is also demonstrated through its support to object recognition task.

The recent availability of high-resolution multiview ZY-3 satellite images, with angular information, can provide an opportunity to capture 3-D structural features for classification. In high-resolution image classification over urban areas, objects with diverse vertical structures make urban landscape more heterogeneous in 3-D space and consequently can make the classification challenging. In this article, a novel multiangle gray-level cooccurrence tensor feature is proposed based on the multiview bands of the ZY-3 imagery, namely, GLCMMA–T. The GLCMMA–Tfeature captures the distributions of the gray-level spatial variation under different viewing angles, which can depict the 3-D textures and structures of urban objects. The spectral and GLCMMA–Ttensor features are interpreted by two 3-D convolutional neural network (CNN) streams and then concatenated as the input to the fully connected layer. This novel multispectral and multiangle 3-D convolutional neural network (M2-3-DCNN) combines the spectral and angular information, and the fused feature has the potential to provide a comprehensive description of urban objects with complex vertical structures. The experimental results on ZY-3 multiview images from four test areas indicate that the proposed method can significantly improve the classification accuracy when compared with several state-of-the-art multiangle features and deep-learning-based image classification methods.

In the hyperspectral image, each pixel corresponds to a small area on the Earth's surface and represents the intrinsic characteristic of objects, which can be applied for recognition of land covers. Nevertheless, hyperspectral image processing should face some critical issues, and a small sample set problem may be the most challenging one in the research. Deep learning (DL), which has successfully been applied in many fields, has also been introduced for hyperspectral image classification. However, the large gap between the massive parameters to be tuned and limited labeled samples can lead to overfitting scenario, inevitably deteriorating the generalization ability of the DL model. In this article, a lightweight convolutional neural network (LWCNN) is proposed for hyperspectral image classification to mainly tackle the small sample set problem. Especially, spatial-spectral Schroedinger eigenmaps (SSSE) feature extraction is first adopted to obtain the joint spatial-spectral information, and the compressed dimensionality could significantly reduce the number of parameters in the following DL model. Second, a dual-scale convolution (DSC) module is carefully designed to address the SSSE features from a 1-D vector viewpoint (the number of parameters is further decreased), and the DSC procedure is successively employed to obtain the hierarchical structure description that could represent data distribution from different aspects. Subsequently, the feature vectors from all DSC layers are separately filtered by a new bichannel fusion (BCF) module, which could well encode both the intrinsic and contextual information inside DSC features. Finally, the filtered features are concatenated together and imported into a global average pooling classifier to achieve the predicted probability of each category. Experimental results on three famous hyperspectral image data sets illustrate that the developed LWCNN approach is advantageous in both the efficiency and robustness sides for hyperspectral image classification tasks and outperforms other state-of-the-art methods (both traditional-based and DL-based) with very limited labeled samples.

The rapid increase in the number of remote sensing sensors makes it possible to develop multisource feature extraction and fusion techniques to improve the classification accuracy of surface materials. It has been reported that light detection and ranging (LiDAR) data can contribute complementary information to hyperspectral images (HSIs). In this article, a multiple feature-based superpixel-level decision fusion (MFSuDF) method is proposed for HSIs and LiDAR data classification. Specifically, superpixel-guided kernel principal component analysis (KPCA) is first designed and applied to HSIs to both reduce the dimensions and compress the noise impact. Next, 2-D and 3-D Gabor filters are, respectively, employed on the KPCA-reduced HSIs and LiDAR data to obtain discriminative Gabor features, and the magnitude and phase information are both taken into account. Three different modules, including the raw data-based feature cube (concatenated KPCA-reduced HSIs and LiDAR data), the Gabor magnitude feature cube, and the Gabor phase feature cube (concatenation of the corresponding Gabor features extracted from the KPCA-reduced HSIs and LiDAR data), can be, thus, achieved. After that, random forest (RF) classifier and quadrant bit coding (QBC) are introduced to separately accomplish the classification task on the aforementioned three extracted feature cubes. Alternatively, two superpixel maps are generated by utilizing the multichannel simple noniterative clustering (SNIC) and entropy rate superpixel segmentation (ERS) algorithms on the combined HSIs and LiDAR data, which are then used to regularize the three classification maps. Finally, a weighted majority voting-based decision fusion strategy is incorporated to effectively enhance the joint use of the multisource data. The proposed approach is, thus, named MFSuDF. A series of experiments are conducted on three real-world data sets to demonstrate the effectiveness of the proposed MFSuDF approach. The experimental results show that our MFSuDF can achieve the overall accuracy of 73.64%, 93.88%, and 74.11% for Houston, Trento, and Missouri University and University of Florida (MUUFL) Gulport data sets, respectively, when there are only three samples per class for training.

This article presents a novel adaptive edge preserving (aEP) scheme in Markov random fields (MRFs) for hyperspectral image (HSI) classification. MRF regularization usually suffered from over-smoothing at boundaries and insufficient refinement within class objects. This work divides and conquers this problem class-by-class, and integrates${K}$(${K} -1$)/2 (${K}$is the class number) aEP maps (aEPMs) in MRF model. Spatial label dependence measure (SLDM) is designed to estimate the interpixel label dependence for given spectral similarity measure. For each class pair, aEPM is optimized by maximizing the difference between intraclass and interclass SLDM. Then, aEPMs are integrated with multilevel logistic (MLL) model to regularize the raw pixelwise labeling obtained by spectral and spectral–spatial methods, respectively. The graph-cuts-based$\alpha ~\beta $-swap algorithm is modified to optimize the designed energy function. Moreover, to evaluate the final refined results at edges and small details thoroughly, segmentation evaluation metrics are introduced. Experiments conducted on real HSI data denote the superiority of aEPMs in evaluation metrics and region consistency, especially in detail preservation.

In many remote sensing tasks, different types of regions or targets differ in requirements for spectral and spatial quality. The discrepancy reveals that a uniform pansharpening strategy applying to the entire image may not fulfill the varying demands of different regions appropriately. From this aspect, we resort to saliency analysis to distinguish regions with different spatial and spectral requirements and then propose a new saliency cascade convolutional neural network for pansharpening (SC-PNN). SC-PNN is composed of two parts: a dilated deformable convolutional network (DDCN) for saliency analysis and a saliency cascade residual dense network (SC-RDN) for pansharpening. DDCN is a fully convolutional network based on hybrid dilated convolution and deformable convolution, aiming to separate salient regions, such as residential areas from nonsalient areas, including mountains and vegetation areas, with well-defined boundaries and integrity. In the fusion process, SC-RDN is specially designed with the help of saliency analysis. We first construct a deep regression network to estimate a primarily sharpened image and subsequently leverage the saliency map produced by DDCN to develop a saliency enhancement module. In this module, the quality of salient and nonsalient areas is further improved by two independent deep residual dense networks. Thus, a precise fused image can be predicted. Experiments on SPOT5, GeoEye-1, and WorldView-3 data sets reveal that, compared to state-of-the-art pansharpening methods, our proposal has a superior ability to improve the spatial quality and preserve spectral information. The effectiveness of the saliency enhancement module is also validated in the experiment.

This article presents an intrafusion network (IFN) for hyperspectral image (HSI) super-resolution (SR). Given that the HSI is a 3-D data cube with both the spatial information and the spectral information, the key challenge to construct HSI SR is how to efficiently exploit the spectral information among consecutive low-resolution (LR) bands, besides the spatial information. The proposed IFN consists of three modules, including the spectral difference module, the parallel convolution module, and the intrafusion module, which directly utilizes both the spatial information and the spectral information for reconstructing the high-resolution HSI. Different from most of the existed methods that tackle the spatial and spectral information separately, the proposed spatial-spectral utilization is achieved in one integrated network, which opens up a new way for HSI SR. Meanwhile, applications of this three modules strategy (first spectral difference, then parallel convolution, and finally, intrafusion) on both the conventional convolutional neural network and the residual network with deeper depth have shown the generalization capacity of this proposal. Experimental results and data analysis demonstrate the effectiveness of the proposed method using three hyperspectral data sets.

As a typical graph-based semisupervised learning technique, the label propagation (LP) approach has gained much attention in recent years. The key to LP algorithms is the propagation capability and efficiency of the similarity matrix, which describes the similarity between two data points. Concerning hyperspectral image which often contains hundreds of thousands of pixels, the corresponding similarity matrix is particularly huge and thus the LP procedure is intractable. Fortunately, superpixel, which can effectively characterize the spatial semantic information of surface objects, provides a reasonable way to solve this problem. In this article, we propose an elaborate superpixel-based weighted LP approach, abbreviated as SuWLP, for hyperspectral image classification. First, the hyperspectral image is oversegmented by the entropy rate segmentation (ERS) method, and the internal consistency of each superpixel can be achieved. Second, a new similarity measure is designed to estimate the similarity between two superpixels, and a superpixel-based similarity matrix can be thus established. Third, after the training samples have been expanded based on the superpixel distribution, a weighted LP technique is designed to propagate the sample label at the superpixel level without any parameter tuning. Finally, the label of each superpixel maps back to the contained pixels. We compared our proposed SuWLP method with several state-of-the-art ones, and experimental results on three real hyperspectral data sets certify the effectiveness and efficiency of the superpixel-level LP strategy.

Detecting moving objects from ground-based videos is commonly achieved by using background subtraction (BS) techniques. Low-rank matrix decomposition inspires a set of state-of-the-art approaches for this task. It is integrated with structured sparsity regularization to achieve BS in the developed method of low-rank and structured sparse decomposition (LSD). However, when this method is applied to satellite videos where spatial resolution is poor and targets' contrast to the background is low, its performance is limited as the data no longer fit adequately either the foreground structure or the background model. In this article, we handle these unexplained data explicitly and address the moving target detection from space as one of the pioneering studies. We propose a new technique by extending the decomposition formulation with bounded errors, named Extended LSD (E-LSD). This formulation integrates low-rank background, structured sparse foreground, as well as their residuals in a matrix decomposition problem. Solving this optimization problem is challenging. We provide an effective solution by introducing an alternative treatment and adopting the direct extension of alternating direction method of multipliers (ADMM). The proposed E-LSD was validated on two satellite videos, and the experimental results demonstrate the improvement in background modeling with boosted moving object detection precision over state-of-the-art methods.

Inspired by the recent developments in computer vision, low-rank and structured sparse matrix decomposition can be potentially be used for extract moving objects in satellite videos. This set of approaches seeks for rank minimization on the background that typically requires batch-based optimization over a sequence of frames, which causes delays in processing and limits their applications. To remedy this delay, we propose an online low-rank and structured sparse decomposition (O-LSD). O-LSD reformulates the batch-based low-rank matrix decomposition with the structured sparse penalty to its equivalent framewise separable counterpart, which then defines a stochastic optimization problem for online subspace basis estimation. In order to promote online processing, O-LSD conducts the foreground and background separations and the subspace basis update alternatingly for every frame in a video. We also show the convergence of O-LSD theoretically. Experimental results on two satellite videos demonstrate the performance of O-LSD in terms of accuracy, and the time consumption is comparable with the batch-based approaches with significantly reduced delay in processing.

Spatial identification of urban-rural fringes (URF) is crucial for monitoring urban sprawl and mapping out urban management planning. This paper proposes an efficient approach for extracting URF, integrating optical and nighttime lights (NTL) data. Results illustrate that the proposed approach is an effective and practical algorithm for URF identification. The findings highlight the potential of combining optical and NTL data for earth observation, which provide opportunities for new applications.

Automatic grass weed detection and classification deals with large intraclass challenges since they are similar to grass in shape, sizes and colors. Grass weed detection and mapping is critical for site-specific weed control to reduce the cost of labor and impact of herbicides. In this paper, we investigate different shape and texture based feature extraction methods to discriminate three different grass weed categories using hyperspectral images. Feature extraction methods including Gabor features, Histogram of Oriented Gradients (HoG), and Local Binary Pattern (LBP) are evaluated in this paper. The experimental results indicate that the overall accuracy of the grass weed classification using the propose combination of texture and shape features generated from LBP and HoG is higher as compared to using shape and texture features separately.

With supervised machine learning algorithms, meaningful information can be extracted from satellite images to support rice farm mapping. The success of these algorithms depends largely on the availability and quality of ground-truth reference data. However, collecting such data is often laborious and time-consuming. The fast development of drone technique has opened up an efficient way for ground truthing. In this study, we construct a drone-based sensing system for the purpose of efficiently collecting ground-truth data. The drone carries dual cameras that provide multispectrsal images with high spatial resolution in the same sensed site. The system is dedicated to collecting training data for rice farm mapping in Australia. To demonstrate the ability of the constructed system, real-flight experiments were conducted. Drone images of rice crops were acquired in the Riverina region of Australia, during the 2018-2019 summer season. Three-dimensional models were constructed from multiple images captured by the drone, where the structural information of rice crops was extracted. Results show that the configuration of dual cameras and the construction of three-dimensional models are particularly advantageous for ground truthing, providing valuable information for reliably identifying rice crops.

Detecting moving objects from satellite videos decomposes video frames into low rank background with additive sparse foreground, and moving objects are commonly rec-ognized by pixel-wise sparsity in the foreground. In prac-tice, moving objects are sets of spatially related pixels and tend to present structured sparsity rather than pixel-wise sparsity. Based on this spatial prior, Structured Sparsity Inducing Norm models moving objects as sparse sets of neighboring pixels, however, the isolated sparsity, which is unsuited to the spatial prior, is left to corrupt the low rank background. In order to address the corruption of unstructured sparsity to background, we proposed a new decomposition formulation named Improved Low Rank plus Structured Sparsity and Unstructured Sparsity Decomposition (ILRSUSD). An inexact Alternating Direction Method is then proposed to solve the improved decomposition formulation efficiently. Experimental result on a satellite video dataset demonstrates our improvement in background modeling with boosted moving object detection precision against state-of-art approaches.

Weed detection and classification are one of the important and crucial steps for area-specific weed control. This reduces the overall cost and the negative impact of using unnecessary herbicides on human health and crops. As the spectral similarity between weeds and crops is high, patch-based classification approaches are adopted in this letter. Convolutional neural network (CNN) and histogram of oriented gradients (HoG) methods are evaluated and compared. With the advancement in the remote sensing technologies, a large number of sensors are available which provide different number of bands with different spatial resolutions. The effect of these variations on weed identification is investigated. Experimental results show that CNN method extracts more discriminative and powerful features that lead to an accurate classification of different weeds compared to the HoG method. Analysis of the two important parameters provides guidance in choosing correct patch size, spatial resolution, and the number of bands to use when CNN is applied for accurate and efficient weed classification.

Image classification is considered to be one of the critical tasks in hyperspectral remote sensing image processing. Recently, a convolutional neural network (CNN) has established itself as a powerful model in classification by demonstrating excellent performances. The use of a graphical model such as a conditional random field (CRF) contributes further in capturing contextual information and thus improving the classification performance. In this paper, we propose a method to classify hyperspectral images by considering both spectral and spatial information via a combined framework consisting of CNN and CRF. We use multiple spectral band groups to learn deep features using CNN, and then formulate deep CRF with CNN-based unary and pairwise potential functions to effectively extract the semantic correlations between patches consisting of 3-D data cubes. Furthermore, we introduce a deep deconvolution network that improves the final classification performance. We also introduced a new data set and experimented our proposed method on it along with several widely adopted benchmark data sets to evaluate the effectiveness of our method. By comparing our results with those from several state-of-the-art models, we show the promising potential of our method.

Spatial information has been demonstrated to be useful for hyperspectral images (HSIs) classification. The challenge is that spatial properties are often present at various spatial scales instead of a single fixed scale. A multiscale conservative smoothing algorithm is proposed in this paper to reduce noise and extract spatial structure information from coarse to fine levels. Over-smoothing is prevented automatically by imposing a weighting scheme on the neighboring pixels used for smoothing, where dissimilar neighbors' contributions are suppressed. Motived by multitask learning, an adaptive sparse representation is introduced to integrate different characteristics from the series of enhanced HSIs. The sparse coefficients of a given unknown pixel can be obtained from this representation and then used for classification. Experiments conducted on three benchmark data sets demonstrate that the proposed methodology leads to superior classification performance when compared to several well-known classifiers.

In virtue of the spatial structural characteristic of surface materials, the performance of the hyperspectral image classification can be boosted by incorporating texture information. Normally, the spatial structure can be extracted by predefined operators, including the popular extended multiattribute profiles (EMAPs) and the Gabor filters. Recently, superpixel segmentation, which reflects the homogeneous regularity of objects, has drawn much attention in the field. In this paper, a collaborative representation-based multiscale superpixel fusion (CRMSF) approach has been proposed for the hyperspectral image classification. First, after obtaining the EMAPs from the raw hyperspectral image, a group of predesigned 3-D Gabor wavelet filters is convolved with the EMAP features, and the EMAP-Gabor features can, thus, be achieved. Second, the collaborative representation-based classification (CRC) is employed to fully and efficiently make use of the huge amount of extracted EMAP-Gabor features. Third, multiscale superpixel maps are generated from the EMAP features that are utilized to regularize the classification map obtained by CRC. A heuristic strategy has been specially devised to automatically decide the number of extracted superpixels in multiple scales, which can be perfectly compatible with hyperspectral images having various spatial sizes and spatial resolutions. This is the most important contribution of the developed CRMSF approach. Finally, the classification task is accomplished by fusing the multiple regularized classification maps. The CRMSF approach has been evaluated on four popular hyperspectral image data sets, and the experimental results show the advantages of CRMSF, particularly for a hyperspectral image with high spatial resolution.

Since the spatial distribution of surface materials is usually regular and locally continuous, it is reasonable to utilize the spectral and spatial information for the hyperspectral image classification. In this paper, a spectral-spatial Gabor surface feature (GSF) fusion approach has been proposed for hyperspectral image classification. First, Gabor magnitude pictures (GMPs) are extracted by applying a set of predefined 2-D Gabor filters to hyperspectral images. Second, the GSF has been extended to the spectral-spatial domains to comply with the 3-D structure of hyperspectral imagery, called 3-DGSF, which utilizes the first-order derivative of GMPs. Meanwhile, a classic superpixel segmentation method, called simple linear iterative clustering (SLIC), is adopted to divide the original hyperspectral image into disjoint superpixels. Third, principal component analysis is adopted to reduce the dimensionality of each extracted 3-DGSF feature cube. Next, a support vector machine classifier is applied on each reduced 3-DGSF features, and the majority voting strategy is used to obtain the classification results. Finally, the superpixel map obtained by SLIC is used to regularize the classification map, and thus, the proposed approach is named as S3-DGSF. Extensive experiments on three real hyperspectral data sets have demonstrated the higher performance of the proposed S3-DGSF approach over several state-of-the-art methods in the literature.

Hyperspectral remote sensing imagery provides valuable and rich information to distinguish the characteristics of materials. However, this advantage of hyperspectral imagery often encounters the problem of a limited amount of training samples, which is caused by the difficulty of manually labeling. Fortunately, the spatial distribution of surface objects can be integrated with the spectral signature to improve the discriminative ability. In this paper, a 3-D Gaussian-Gabor feature extraction and selection framework has been proposed for hyperspectral image classification. First, a bank of 3-D Gaussian-Gabor filters are convolved with the concatenated data of both extended multi-attribute profile (EMAP) features and raw hyperspectral data. Second, an improved fast density peak clustering (IFDPC) method is introduced to select the most representative features from each extracted 3-D Gaussian-Gabor feature cube. Finally, the retained features are combined together to accomplish the classification task. The proposed method is thus named as GG-IFDPC. Three real hyperspectral imagery data sets have been utilized, and the experiments demonstrate the advantages of the proposed GG-IFDPC approach over the compared ones.

Land-cover mapping over urban areas using Landsat imagery has attracted considerable attention in recent years as it can promptly and accurately reflect the biophysical composition status of the urban landscape and allow further applications such as urban planning and risk management. However, due to the large diversity across different urban landscapes, adequate training sample collection for urban area mapping is both challenging and time-consuming. In this paper, we propose a novel unsupervised sample collection method for mapping urban areas using Landsat imagery. Specifically, the idea is to select reliable, representative, and diverse training samples from the images in a two-stage and iterative manner, based on a set of spectral indices (vegetation, impervious surface, soil, water). To validate the effectiveness and robustness of the proposed method, a synthetic data set was designed and a series of Landsat images over 39 representative cities from different biomes across the world was employed. The effectiveness of the proposed algorithm was quantitatively validated by assessing the quality of the automatically collected samples and the accuracy of the mapping results. In terms of the mapping performance, the proposed automatic approach can achieve a comparable mapping accuracy to supervised classification with manually collected samples. On the basis of the freely accessed Landsat data, the proposed approach demonstrates a promising potential for automatic large-scale (i.e., global) mapping over urban areas.

Anomaly detection is one of the most important applications of hyperspectral imaging technology. It is a challenging task due to the high dimensionality of hyperspectral images (HSIs), redundant information, noisy bands, and the limited capability of utilizing spatial information. In this paper, we address these problems and propose a novel anomaly detection method in HSIs. Our approach, called structure tensor and guided filter (STGF)-based strategy for anomaly detection, is based on the characteristics of HSIs. First, a novel band selection algorithm is proposed to reduce dimension, remove noisy bands, and select bands with effective information. Second, the selected bands are decomposed into two parts according to the characteristics of anomalies that are usually in a small area. Followed by this step, the backgrounds are removed through a simple differential operation for each selected band. Considering that not all of the bands provide the same contributions to anomaly detection, we then fuse the differential maps by a novel adaptive weighting method to obtain an initial detection map. Finally, GF is conducted to rectify the previous map under the condition that the neighboring pixels usually have quite strong correlations with each other. Experiments have been conducted on real-scene remote sensing HSI. Comparative analyses validate that the proposed STGF method presents superior performance in terms of detection accuracy and computational time.

Hyperspectral images (HSIs) can describe the subtle differences in the spectral signatures of materials. However, they have low spatial resolution due to various hardware limitations. Improving it via postprocess without an auxiliary high-resolution (HR) image still remains a challenging problem. In this paper, we address this problem and propose a new HSI super-resolution (SR) method. Our approach, called deep feature matrix factorization (DFMF), blends feature matrix extracted by a deep neural network (DNN) with nonnegative matrix factorization strategy for super-resolving real-scene HSI. The estimation of the HR HSI is formulated as a combination of latent spatial feature matrix and spectral feature matrix. In the DFMF model, the input low-resolution (LR) HSI is first partitioned into several subsets according to the correlation matrix, and the key band is selected from each subset. Then, the key band group is super-resolved by a DNN model, and the HR key band group is then used as a guide to carry out deep spatial feature matrix. Specifically, the input LR HSI with prototype reflectance spectral vectors of the scene will be preserved when super-resolving in a spatial domain. Thus, the nonnegative spectral and spatial feature matrices are extracted simultaneously from alternately factorizing the pair of LR HSI and the HR key band group. Finally, the HR HSI is obtained by the integration of the spectral and spatial feature matrices. Experiments have been conducted on real-scene remote sensing HSI. Comparative analyses validate that the proposed DFMF method presents a superior super-resolving performance, as it preserves spectral information better.

The extension of impervious surface serves as a key indicator of urbanization and environmental quality. This paper explores the integrated use of Landsat imagery and nighttime light (NTL) imagery for impervious surface mapping. An optical and NTL based spectral mixture analysis (ON_SMA) at sub-pixel level is proposed where local endmembers are used and adaptive endmember sets are selected based on the information provided by both sensors. Results illustrate that comparing with optical based spectral mixture analysis, the proposed approach yields effective improvement in impervious surface detection due to the contribution of the NTL data. The findings highlight the potential of combining optical and NTL data for earth observation, which provide opportunities for new applications.

Automatic weed detection and mapping are critical for site-specific weed control in order to reduce the cost of farming as well as the impact of herbicides on human health. In this paper, we investigate patch-based weed identification using hyperspectral images. Convolutional Neural Network (CNN) is evaluated and compared with the Histogram of Oriented Gradients (HoG) for this purpose. Suitable patch sizes are investigated. The limitation of RGB imagery is demonstrated. The experimental results indicate that the overall accuracy of the weed classification using CNN increases with the increasing number of bands used. With more bands, CNN extracts more powerful and discriminative features and leads to improved classification as compared to the traditional HoG feature extraction method. The computational load of CNN, however, is slightly increased with the increasing number of bands.

Gaofen-4 is the first optical geostationary satellite in China and it is in operation since 2016. Unfortunately, the mid-infrared band has a Ground Spatial Distance (GSD) of 400 m, which is low and limits its application in temperature retrieval. The calibration of this band is challenging as well. In this paper, a new data fusion method is proposed to improve both image quality and spatial resolution, by making use the advantage of high temporal resolution. Experimental results show that the reconstructed high resolution mid-infrared images not only provide more details visually but also gives more accurate and reliable temperature mapping than the original images. The proposed solution holds potential application for Gaofen-4 in the near future.

The payload Multispectral Instrument (MSI) on the satellite Sentinel- 2A provides data with strong spectral information, reasonable spatial resolution and good revisit time, which make them suitable for crop monitoring. With the availability of the image data over the complete rice growing season over two consecutive years, spectral time series analysis of rice crops is conducted in this study for the Riverina region of Coleambally, New South Wales, Australia. Vegetation and water indices are adopated to compare the growing patterns of rice over the 2015/2016 and 2016/2017 growing seasons. Rice crops of different varieties are identified and examined. Different seed sowing methods are also compared in terms of their effect on the latter season. The results show that the growth pattern of rice follows a particular trend that can be distinguished from other types of vegetation. This is facilitated by a property unique to rice where water sensitive indices produce a higher reading than vegetation indices during the initial flooding period of the season, after which, the crop growth reverses this and the biomass sensitive index becomes larger. Spectral analysis of rice crops planted via various methods reveals that the drill sowing method did not produce this unique characteristic as the late flooding time results in a very short submersion period for the rice seedlings, which is a valuable finding.

Satellite high definition videos provide an opportunity to monitor moving targets over a large territory. However, the low spatial resolution and low contract of these videos make target detecting and tracking a challenging task. In this paper, we propose a zoom-in approach for detecting dim and small target proposals from each single frame of the videos to help with moving target tracking. Initialized by a coarse scale segmentation approach, dim and small targets are embedded in each superpixel due to limited size and weak signals. Similar superpixels are then merged using a graph-based approach based on the measurement of the overlap between their histograms. The background statistics become stronger and target pixels are more obvious in the merged superpixels, so that the target pixels can be extracted. Finally, the corresponding boundary box is generated for each spatially connected target pixels selected inside each superpixel. They form the dim and small target proposals. Experimental results show that our zoom-in scheme can generate less proposals with higher recall rate compared with state-of-the-art proposal extraction algorithms.

Due to the low signal to noise ratio and limited spatial resolution, small target detection in an infrared image is a challenging task. Existing methods often have high false alarm rates and low probabilities of detection when infrared small targets submerge in the background clutter. In this paper, the Convolutional Neural Network (CNN) is adapted to extract the hidden features of small targets from infrared imagery with a proposed technique for a large amount of training data generation. The Point Spread Function (PSF) is employed to model the small target data and generate positive samples. The random background image patches are selected as the negative samples. In this way, the detection problem is skillfully converted into a problem of pattern classification using CNN. Extensive synthetic and real small targets were tested to evaluate the performance of this novel small target detection framework. The experimental results indicate that the proposed algorithm is simple and effective with satisfactory detection accuracy.

Abstract Several fully automated latent print forensic techniques have been reported. In this paper, we propose a fully automated latent print segmentation module for the partition of the fingerprint region in a query latent image, which can help find a corresponding match of the suspect reliably. Being different from the existing methods that build the prelearned dictionary from the high‐quality fingerprint image patches, the proposed dictionary learning procedure is conducted on the target images. The advantages of the proposed method are the following: (i) it does not require a large number of high‐quality “ridge‐valley” atoms and (ii) not only the structure similarity but also the pattern scale has been kept consistent between the target image patches and learned dictionary atoms. Because no commercial latent fingerprint matcher is publicly available and the latent matcher reported in the literature is not accessible to the public either, a latent fingerprint matching platform is implemented to evaluate the obtained segmentation results and automated latent fingerprint matching performance. On the basis of this platform, experimental comparisons are conducted to assess closeness to the system performance upper bound when different segmentation modules are deployed. Moreover, matcher‐independent criteria such as genuine minutiae preservation rate and the segmented region of interest's accuracy are used. All the experimental results demonstrate that the proposed segmentation approach outperforms state‐of‐the‐art techniques in terms of finding the correct fingerprint of the suspect subject.

As a promising signal processing technique, a joint sparse model (JSM) has been used to integrate spatial and spectral information in the classification of remotely sensed images. This technique defines a local region of a fixed window size and assumes an equal contribution from each neighborhood pixel in the classification process of the test pixel. However, equal weighting is less reasonable for heterogeneous pixels, especially around class boundaries. Hence, a discontinuity preserving relaxation (DPR) method can be used to locally smooth the results without crossing the boundaries by detecting the discontinuities of an image in advance. In this letter, we developed a novel strategy that combines these two methods to improve the hyperspectral image classification. A JSM is first applied to obtain a posteriori probability distribution of pixels and then a DPR method is used to further improve the classification results. Experiments conducted on two benchmark data sets demonstrate that the proposed method leads to superior performance when compared with several popular algorithms.

In this letter, the application of super-resolution (SR) techniques to GaoFen(GF)-4, which is the most advanced geostationary-orbit earth observing satellite in China, remote sensing images is investigated and tested. One of the shortcomings of the geostationary-orbit-based earth observing satellite is the limitation of spatial resolution. However, human beings never stop pursuing higher resolution in images. This is the first experiment of applying SR to a sequence of low-resolution (LR) images captured by GF-4 within a short time period. One of the barriers for applying SR to remote sensing images is the large time gaps between those LR image acquisition, because the reflection characteristic of the ground may change within the time period when those LR images were captured. However, GF-4 has the unique advantage of capturing a sequence of LR images of the same region in minutes, i.e., working as a staring camera from the point view of SR. The reconstructed high-resolution images of some regions in Beijing and Hainan are shown and evaluated in this letter. This letter demonstrates that the application of SR to geostationary-orbit-based earth observation data is feasible and valuable, and it has the potential to be applied to the images acquired by all other geostationary-orbit-based earth observing systems.

Balancing exploration and exploitation according to evolutionary states is crucial to meta-heuristic search (M-HS) algorithms. Owing to its simplicity in theory and effectiveness in global optimization, gravitational search algorithm (GSA) has attracted increasing attention in recent years. However, the tradeoff between exploration and exploitation in GSA is achieved mainly by adjusting the size of an archive, named , which stores those superior agents after fitness sorting in each iteration. Since the global property of remains unchanged in the whole evolutionary process, GSA emphasizes exploitation over exploration and suffers from rapid loss of diversity and premature convergence. To address these problems, in this paper, we propose a dynamic neighborhood learning (DNL) strategy to replace the model and thereby present a DNL-based GSA (DNLGSA). The method incorporates the local and global neighborhood topologies for enhancing the exploration and obtaining adaptive balance between exploration and exploitation. The local neighborhoods are dynamically formed based on evolutionary states. To delineate the evolutionary states, two convergence criteria named limit value and population diversity, are introduced. Moreover, a mutation operator is designed for escaping from the local optima on the basis of evolutionary states. The proposed algorithm was evaluated on 27 benchmark problems with different characteristic and various difficulties. The results reveal that DNLGSA exhibits competitive performances when compared with a variety of state-of-the-art M-HS algorithms. Moreover, the incorporation of local neighborhood topology reduces the numbers of calculations of gravitational force and thus alleviates the high computational cost of GSA.

Since it is usually difficult and time-consuming to obtain sufficient training samples by manually labeling, feature extraction, which investigates the characteristics of hyperspectral images (HSIs), such as spectral continuity and spatial locality of surface objects, to achieve the most discriminative feature representation, is very important for HSI classification. Meanwhile, due to the spatial regularity of surface materials, it is desirable to improve the classification performance of HSIs from the superpixel viewpoint. In this paper, we propose a novel local binary pattern (LBP)-based superpixel-level decision fusion method for HSI classification. The proposed framework employs uniform LBP (ULBP) to extract local image features, and then, a support vector machine is utilized to formulate the probability description of each pixel belonging to every class. The composite image of the first three components extracted by a principal component analysis from the HSI data is oversegmented into many homogeneous regions by using the entropy rate segmentation method. Then, a region merging process is applied to make the superpixels obtained more homogeneous and agree with the spatial structure of materials more precisely. Finally, a probability-oriented classification strategy is applied to classify each pixel based on superpixel-level guidance. The proposed framework “ULBP-based superpixel-level decision fusion framework” is named ULBP-SPG. Experimental results on two real HSI data sets have demonstrated that the proposed ULBP-SPG framework is more effective and powerful than several state-of-the-art methods.

A novel band discrimination analysis framework for hyperspectral image (HSI) supervised classification is proposed based on dual density (DD). Different from the popular supervised band selection (BS) approaches which measure the discrimination among classes under multivariate normal distribution hypothesis, our work infers the class discrimination degree (overlapping extent) for valid extraction of band subset without any assumed distribution. In the proposed framework, it is crucial to find indexes to measure the discrimination degree of each band, and therefore we develop the DD indexes, including the homogeneity density and the heterogeneity density. Viewing each band of the HSI as a data set, i.e., the data points in each data set are 1-D, and we first obtain the DD value pairs for all data points in each data set. Then, for each data set, we determine its discrimination degree using DD-based zone ratio or score quantify strategy. Finally, the bands, which are determined as the nonoverlapped or have high scores, are chosen as the band subset for the subsequent classification. Superiorities of the proposed BS are demonstrated on the three real-world HSIs over several well-known BS algorithms in terms of classification accuracy and speed.

In[1], information regarding the corresponding author is missing. The information is updated here. The updated footnote below shows that Xiaoyan Luo is the corresponding author for this paper.

Due to the 3-D nature of hyperspectral images, as well as the spatial properties (such as regularity and continuity) of land covers, many 3-D feature extraction operators have been designed to fully exploit the joint spatial-spectral information. However, the large amount of obtained features can suffer from the “curse of dimensionality” problem, especially for the small training sample set. Moreover, various spatial-spectral features can represent the characteristics of the hyperspectral image from different aspects. In this paper, a multiple 3-D feature fusion framework (M3DF3) has been proposed for hyperspectral image classification. First, we extend the 2-D Gabor surface feature into 3-D (3DSF) domains to comply with the spatial-spectral structure of the hyperspectral image, which is directly applied on the original hyperspectral image instead of the Gabor features. Second, three 3-D feature extraction methods, including the 3-D morphological profile, the 3-D local binary pattern, and the proposed 3DSF, that, respectively, characterize the hyperspectral image from three different angles, i.e., morphology, local dependence, and shape smoothness, are fused under a multitask sparse representation framework to take full advantage of the multiple 3-D features together. The proposed M3DF3approach was fully tested on three real-world hyperspectral image data, i.e., the widely used Indian Pines, Pavia University, and Houston University. The results show that our method can achieve as high as 68.22%, 79.44%, and 72.84% accuracies, respectively, even when only few samples, i.e., three samples per class, are used for training.

The explosive availability of remote sensing images has challenged supervised classification algorithms such as Support Vector Machines (SVM), as training samples tend to be highly limited due to the expensive and laborious task of ground truthing. The temporal correlation and spectral similarity between multitemporal images have opened up an opportunity to alleviate this problem. In this study, a SVM-based Sequential Classifier Training (SCT-SVM) approach is proposed for multitemporal remote sensing image classification. The approach leverages the classifiers of previous images to reduce the required number of training samples for the classifier training of an incoming image. For each incoming image, a rough classifier is firstly predicted based on the temporal trend of a set of previous classifiers. The predicted classifier is then fine-tuned into a more accurate position with current training samples. This approach can be applied progressively to sequential image data, with only a small number of training samples being required from each image. Experiments were conducted with Sentinel-2A multitemporal data over an agricultural area in Australia. Results showed that the proposed SCT-SVM achieved better classification accuracies compared with two state-of-the-art model transfer algorithms. When training data are insufficient, the overall classification accuracy of the incoming image was improved from 76.18% to 94.02% with the proposed SCT-SVM, compared with those obtained without the assistance from previous images. These results demonstrate that the leverage of a priori information from previous images can provide advantageous assistance for later images in multitemporal image classification.

Object proposal approach, where potential objects are extracted first, has been developed to improve the target searching efficiency. In this paper, we investigate two newly developed algorithms, namely Selective Search and EdgeBoxes, for object proposal extraction and evaluate their performance with very high resolution (VHR) remote sensing images. These object proposal extraction methods play a vital role in assisting the target detectors to find the targets rapidly and accurately in an image. Considering that the size of VHR satellite images is large and there are more than one targets in an image, these two methods are highly valuable as they can avoid exhaustive search using a sliding window to find the targets. Both methods were evaluated using NWPU VHR-10 class remote sensing data set. The performance is measured using the recall rate and execution time. Results indicate that the EdgeBoxes method is significantly faster and has high recall rate as compared to the Selective Search method. Therefore, for target detection, it is effective to use EdgeBoxes method for object proposals extraction using VHR satellite images.

Multi-temporal remote sensing imagery has become widely available, which opens up an opportunity to improve the efficiency of supervised classification techniques. While a classifier trained from a previous image (source domain) cannot be directly applied to the current image (target domain) because of changes in imaging conditions and dynamics of land surface spectral properties, domain transfer techniques have been introduced in recent years to remove the need for a complete retraining of the current image data. This approach is further developed in the present study, and a domain transfer algorithm named Temporal-Adaptive Support Vector Machine (TASVM) is proposed. The algorithm enables the adaptation of a classifier trained with the source-domain image to the classification of the target-domain image where class data have a different distribution. The adaptation process is allowed to be conducted at the classifier-level where the source classifiers can be transferred without re-accessing the source domain raw data. Experimental analysis showed that the proposed algorithm generated stable results, especially under the circumstances where satisfactory results were hard to achieve with traditional algorithms.

The multisensory fusion of remote sensing data has obtained a great attention in recent years. In this letter, we propose a new feature fusion framework based on deep neural networks (DNNs). The proposed framework employs deep convolutional neural networks (CNNs) to effectively extract features of multi-/hyperspectral and light detection and ranging data. Then, a fully connected DNN is designed to fuse the heterogeneous features obtained by the previous CNNs. Through the aforementioned deep networks, one can extract the discriminant and invariant features of remote sensing data, which are useful for further processing. At last, logistic regression is used to produce the final classification results. Dropout and batch normalization strategies are adopted in the deep fusion framework to further improve classification accuracy. The obtained results reveal that the proposed deep fusion model provides competitive results in terms of classification accuracy. Furthermore, the proposed deep learning idea opens a new window for future remote sensing data fusion.

Recently, the capability of deep learning-based approaches, especially deep convolutional neural networks (CNNs), has been investigated for hyperspectral remote sensing feature extraction (FE) and classification. Due to the large number of learnable parameters in convolutional filters, lots of training samples are needed in deep CNNs to avoid the overfitting problem. On the other hand, Gabor filtering can effectively extract spatial information including edges and textures, which may reduce the FE burden of the CNNs. In this letter, in order to make the most of deep CNN and Gabor filtering, a new strategy, which combines Gabor filters with convolutional filters, is proposed for hyperspectral image classification to mitigate the problem of overfitting. The obtained results reveal that the proposed model provides competitive results in terms of classification accuracy, especially when only a limited number of training samples are available.

With the rapid development of spectral imaging techniques, classification of hyperspectral images (HSIs) has attracted great attention in various applications such as land survey and resource monitoring in the field of remote sensing. A key challenge in HSI classification is how to explore effective approaches to fully use the spatial-spectral information provided by the data cube. Multiple kernel learning (MKL) has been successfully applied to HSI classification due to its capacity to handle heterogeneous fusion of both spectral and spatial features. This approach can generate an adaptive kernel as an optimally weighted sum of a few fixed kernels to model a nonlinear data structure. In this way, the difficulty of kernel selection and the limitation of a fixed kernel can be alleviated. Various MKL algorithms have been developed in recent years, such as the general MKL, the subspace MKL, the nonlinear MKL, the sparse MKL, and the ensemble MKL. The goal of this paper is to provide a systematic review of MKL methods, which have been applied to HSI classification. We also analyze and evaluate different MKL algorithms and their respective characteristics in different cases of HSI classification cases. Finally, we discuss the future direction and trends of research in this area.

Considering that satellites rarely acquire data from the exact nadir direction, angular effect normalization needs to be conducted as an important preprocessing step to correct reflectance observations from off-nadir directions into the nadir direction. Kernel-based bidirectional reflectance distribution function models have been employed for angular effect correction. The kernels used in the model are often predetermined and fixed for an entire image. However, the fixed kernels are unable to accommodate the various reflective characteristics of different ground cover types present in the imaged area. In this paper, we propose a kernel learning procedure that enables the flexible selection of kernels for different land cover types within a scene. The kernels are selected from kernel dictionaries that contain multiple candidate kernels. The selection is conducted on the superpixel level instead of the pixel level in order to reduce within-class variation and overcome the overfitting problem. Experiments are conducted on multiangular images acquired by the Sentinel-2A satellite over a rural area in southeastern Australia. Cross-validation results show that the proposed method is able to adaptively select appropriate kernels for different land cover types, leading to an improved performance for image normalization.

Due to the high spectral dimensionality of hyperspectral images as well as the difficult and time-consuming process of collecting sufficient labeled samples in practice, the small sample size scenario is one crucial problem and a challenging issue for hyperspectral image classification. Fortunately, the structure information of materials, reflecting region of homogeneity in the spatial domain, offers an invaluable complement to the spectral information. Assuming some spatial regularity and locality of surface materials, it is reasonable to segment the image into different homogeneous parts in advance, called superpixel, which can be used to improve the classification performance. In this paper, a superpixel-based multitask learning framework has been proposed for hyperspectral image classification. Specifically, a set of 2-D Gabor filters are first applied to hyperspectral images to extract discriminative features. Meanwhile, a superpixel map is generated from the hyperspectral images. Second, a superpixel-based spatial-spectral Schroedinger eigenmaps (S4E) method is adopted to effectively reduce the dimensions of each extracted Gabor cube. Finally, the classification is carried out by a support vector machine (SVM)-based multitask learning framework. The proposed approach is thus termed Gabor S4E and SVM-based multitask learning (GS4E-MTLSVM). A series of experiments is conducted on three real hyperspectral image data sets to demonstrate the effectiveness of the proposed GS4E-MTLSVM approach. The experimental results show that the performance of the proposed GS4E-MTLSVM is better than those of several state-of-the-art methods, while the computational complexity has been greatly reduced, compared with the pixel-based spatial-spectral Schroedinger eigenmaps method.

The local binary pattern (LBP) is a simple and efficient texture descriptor for image processing. Recently, LBP has been introduced for feature extraction of hyperspectral imagery. Specifically, the LBP codes are extracted from the 2-D band images to capture the spatial correlation among neighboring pixels, and then the statistical histogram features from all bands, which could estimate the underlying distribution in local area, are concatenated together for pixel-wise classification. However, since hyperspectral imagery contains rich spectral and spatial information, which is actually a 3-D data cube, the 2-D LBP (2-DLBP) model cannot fully exploit the joint spectral-spatial structure. In this paper, the 2-DLBP has been extended into 3-D LBP (3-DLBP) model through forming a 3-D regular octahedral frame to characterize the spectral-spatial relationship. In order to reflect the local continuous property of hyperspectral data in both the spectral and spatial domains, while ensuring the rotational invariance of the 3-DLBP model, the code patterns of 3-DLBP model have been divided into eight groups (including seven groups of “dense” patterns and one group of “nondense” patterns) based on the consistency of spectral-spatial topology structure. Specifically, the patterns in seven “dense” groups correspond to the microstructures in the 3-D domains (such as spots, edges, and flat areas), which has a high percentage in all the 3-DLBP patterns, while the rest patterns are aggregated and treated as the “nondense” patterns. The proposed method is thus called 3-D dense LBP (3-D2LBP) model. Moreover, instead of taking zero as the hard threshold, a slack variable has been introduced to enable the difference between the central pixel and the neighboring ones varying in a small interval, which could greatly decrease the impact of spectral variability and noise, and the discriminative power of the features has been further boosted. The slack threshold-based 3-D2LBP model is named ST-3-D2LBP. A series of experiments is conducted on three real hyperspectral imageries to demonstrate the effectiveness of the proposed two 3-D2LBP-based methods. The experimental results show that the performance of the proposed ST-3-D2LBP is significantly superior to that of 2-DLBP, which is also better than the 3-D2LBP model and several state-of-the-art hyperspectral classification methods.

In this paper, a new framework of mixed sparse representations (MSRs) is proposed for solving ill-conditioned problems with remote sensing images. In general, it is very difficult to find a common sparse representation for remote sensing images because of complicated ground features. Here we regard a remote sensing image as a combination of subimage of smooth, edges, and point-like components, respectively. Since each domain transformation method is capable of representing only a particular kind of ground object or texture, a group of domain transformations are used to sparsely represent each subimage. To demonstrate the effect of the framework of MSR for remote sensing images, MSR is regarded as a prior for maximum a posteriori when solving ill-conditioned problems such as classification and super resolution (SR), respectively. The experimental results show that not only the new framework of MSR can improve classification accuracy but also it can construct a much better high-resolution image than other common SR methods. The proposed framework MSR is a competitive candidate for solving other remote sensing images-related ill-conditioned problems.

Supervised classification of hyperspectral images (HSI) is a very challenging task due to the existence of noisy and mixed spectral characteristics. Recently, the widely developed spectral unmixing techniques offer the possibility to extract spectral mixture information at a subpixel level, which can contribute to the categorization of seriously mixed spectral pixels. Besides, it has been demonstrated that the discrimination between different materials will be improved by integrating the geometry and structure information, which can be derived from the variance between neighboring pixels. Furthermore, by incorporating the spatial context, the superpixel-based spectral-spatial similarity information can be used to smooth classification results in homogeneous regions. Therefore, a novel fusion framework for HSI classification that combines subpixel, pixel, and superpixel-based complementary information is proposed in this paper. Here, both feature fusion and decision fusion schemes are introduced. For the feature fusion scheme, the first step is to extract subpixel-level, pixel-level, and superpixel-level features from HSI, respectively. Then, the multiple feature-induced kernels are fused to form one composite kernel, which is incorporated with a support vector machine (SVM) classifier for label assignment. For the decision fusion scheme, class probabilities based on three different features are estimated by the probabilistic SVM classifier first. Then, the class probabilities are adaptively fused to form a probabilistic decision rule for classification. Experimental results tested on different real HSI images can demonstrate the effectiveness of the proposed fusion schemes in improving discrimination capability, when compared with the classification results relied on each individual feature.

Less training samples are a challenging problem in hyperspectral image classification. Active learning and semisupervised learning are two promising techniques to address the problem. Active learning solves the problem by improving the quality of the training samples, while semisupervised learning solves the problem by increasing the quantity of the training samples. However, they pay too much attention to the discriminative information in the unlabeled data, leading to information bias to train supervised models, and much more effort to label samples. Therefore, a method to discover representativeness and discriminativeness by semisupervised active learning is proposed. It takes advantages of both active learning and semisupervised learning. The representativeness and discriminativeness are discovered with a labeling process based on a supervised clustering technique and classification results. Specifically, the supervised clustering results can discover important structural information in the unlabeled data, and the classification results are also highly confidential in the active-learning process. With these clustering results and classification results, we can assign pseudolabels to the unlabeled data. Meanwhile, the unlabeled samples that cannot be assigned with pseudolabels with high confidence at each iteration are regarded as candidates in active learning. The methodology is validated on four hyperspectral data sets. Significant improvements in classification accuracy are achieved by the proposed method with respect to the state-of-the-art methods.

A novel segment-oriented dictionary learning (SeODL) framework for hyperspectral image (HSI) classification is proposed. Differing from existing HSI classification methods which directly process the original whole spectral curves of pixels, our work focuses on local segment analysis to achieve fine depiction and effective exploitation. Viewing the separated segment as a basic processing unit, we first cluster them into two sets with the homogeneity in trend and fluctuation, and then two small dictionaries can be quickly learned. Second, to get meticulous and discriminability enhanced segment-oriented representations (SORs), the segments of the training and test pixels are coded on a novel binary-separated coding strategy. The coding stage for obtaining SORs is sped up by the employment of our proposed enhanced orthogonal matching pursuit technique. A characteristic splicing classifier with high performance can be trained using these SORs of the training pixels. Finally, a spiral searching strategy and a multiple majority-voting method are adopted for fully spatial information incorporation of the test pixels whose final SORs will be embedded into the trained characteristics splicing classifier to ascertain the labels. Experimental results on three real HSI data sets demonstrate the superiority of the proposed SeODL framework over several well-known classification algorithms in terms of classification accuracies.

This paper proposes a method that uses both spectral and spatial information to segment remote sensing hyperspectral images. After a hyperspectral image is over-segmented into superpixels, a deep Convolutional Neural Network (CNN) is used to perform superpixel-level labelling. To further delineate objects from a hyperspectral scene, this paper attempts to combine the properties of CNN and Conditional Random Field (CRF). A mean-field approximation algorithm for CRF inference is used and formulated with Gaussian pairwise potentials as Recurrent Neural Network. This combined network is then plugged into the CNN which leads to a deep network that has robust characteristics of both CNN and CRF. Preliminary results suggest the usefulness of this framework to a promising extent.

The existing kernel-based methods for retrieving land surface bidirectional reflectance distribution functions (BRDFs) usually use a pre-determined combination of kernels and it is fixed for an entire image, which is unable to accommodate the different characteristics from various land cover types. In this study, a multi-kernel method based on l1-norm optimization is proposed. This method is able to automatically select appropriate kernels for each pixel from a kernel dictionary that contains several commonly used kernels. Experimental results show that BRDF retrieval accuracy is improved by adopting this new method.

Local binary patterns (LBP) features extracted from hyperspectral imagery (HSI) have gained impressive performance in hyperspectral classification tasks, for which LBP got considerable attention. However, existing LBP-based hyperspectral imagery classification methods utilized two-dimensional LBP (2DLBP) that could capture gray variation signal in space, which did not excavate the contextual information that hides in spectral-spatial structure considering that hyperspectral imagery characterizes by three dimension. Aimed at this problem, this paper presents a three-dimensional LBP-based (3DLBP) hyperspectral imagery classification method where 2DLBP textures histogram on three orthogonal planes are concatenated to form 3DLBP texture features to be classified by sparse representation. A serial of experiments are conducted on the Pavia university dataset, and the experimental results show that the performance of 3DLBP is significantly superior to that of 2DLBP.

Sparse representation-based classification (SRC) assigns a test sample to the class with minimal representation error via a sparse linear combination of all the training samples, which has successfully been applied to hyperspectral imagery (HSI). Meanwhile, spatial information, that means the adjacent pixels belong to the same class with a high probability, is a valuable complement to the spectral information. In this paper, we propose an efficient method for HSI classification by using superpixel based sparse representation-based classification (SP-SRC). One superpixel can be regarded as a small region consisting of a number of pixels with similar spectral characteristics. The novel method utilizes superpixel to exploit spatial information which can greatly improve classification accuracy. Specifically, SRC is firstly used to classifier the HSI. Then an efficient segmentation algorithm is adopted to divide the HSI into disjoint superpixels. Finally, each superpixel is used to fuse the results of the SRC classifier. Experimental results on the widely-used Indian Pines hyperspectral imagery have shown that the proposed SP-SRC approach could achieve better performance than the pixel-wise SRC method.

Optical remote sensing images are often contaminated by smoke from forest fires or biomass burning. In this paper, a smoke removal method is proposed based on a spectral unmixing technique. Smoke components are detected first by generating subpixel smoke fraction masks with spectral mixture analysis. The smoke component is then subtracted from each smoky pixel. Finally, the attenuated signal is restored by rescaling the abundances of the endmember classes present in each pixel. The proposed method has high feasibility and is not dependent on other auxiliary data. The experiments have been conducted on an AVIRIS data set and the results show that the proposed method is effective for smoke detection and data correction.

Several spectral unmixing techniques using multiple endmembers for each class have been developed. Although they can address within-class spectral variability, their unmixing results may have low unmixing resolution when the within-class variation is large due to the associated high uncertainty. Therefore, it is critical to represent data in an effective feature space so that the endmember classes are compact with small variation. In this letter, a minimum-class-variance support vector machine (MCVSVM) is further developed to extend its functions for both classification and spectral unmixing. Moreover, analytical expressions for spectral unmixing resolution (SUR) are provided to measure the spectral unmixing uncertainty in the new feature space. The extended MCVSVM (e_MCVSVM) can improve SUR and reduce the spectral unmixing uncertainty as it can effectively maximize the between-class scatter while minimizing the within-class scatter. Experimental results show that the e_MCVSVM algorithm performs better in terms of the unmixing accuracy and the computation speed compared with the other algorithms (e.g., fully constrained least squares and endmember bundles) in both linearly separable and nonseparable cases. This newly proposed approach advances the linear spectral mixture analysis with greater speed and higher accuracy based on the SVM after the SUR is effectively characterized.

The multilayer network in deep learning provides a promising means for rich data representation. Inspired by this approach, we investigate multilayer unmixing for spectral decomposition with fast kernel archetypal analysis (KAA). KAA is used for endmember extraction and abundance estimation simultaneously. To refine the initial unmixing results, a multilayer process is utilized to provide final unmixing results at the end of the network. Moreover, a fast implementation of KAA is proposed via using the Nyström method to relieve KAA's memory issue and decrease the processing time. The proposed method is tested on both synthetic and real hyperspectral image data sets. The results demonstrate that the multilayer unmixing algorithm outperforms the conventional unmixing techniques.

Due to the advantages of deep learning, in this paper, a regularized deep feature extraction (FE) method is presented for hyperspectral image (HSI) classification using a convolutional neural network (CNN). The proposed approach employs several convolutional and pooling layers to extract deep features from HSIs, which are nonlinear, discriminant, and invariant. These features are useful for image classification and target detection. Furthermore, in order to address the common issue of imbalance between high dimensionality and limited availability of training samples for the classification of HSI, a few strategies such as L2 regularization and dropout are investigated to avoid overfitting in class data modeling. More importantly, we propose a 3-D CNN-based FE model with combined regularization to extract effective spectral-spatial features of hyperspectral imagery. Finally, in order to further improve the performance, a virtual sample enhanced method is proposed. The proposed approaches are carried out on three widely used hyperspectral data sets: Indian Pines, University of Pavia, and Kennedy Space Center. The obtained results reveal that the proposed models with sparse constraints provide competitive results to state-of-the-art methods. In addition, the proposed deep FE opens a new window for further research.

In this paper, we propose a novel multiple kernel learning (MKL) framework to incorporate both spectral and spatial features for hyperspectral image classification, which is called multiple-structure-element nonlinear MKL (MultiSE-NMKL). In the proposed framework, multiple structure elements (MultiSEs) are employed to generate extended morphological profiles (EMPs) to present spatial-spectral information. In order to better mine interscale and interstructure similarity among EMPs, a nonlinear MKL (NMKL) is introduced to learn an optimal combined kernel from the predefined linear base kernels. We integrate this NMKL with support vector machines (SVMs) and reduce the min-max problem to a simple minimization problem. The optimal weight for each kernel matrix is then solved by a projection-based gradient descent algorithm. The advantages of using nonlinear combination of base kernels and multiSE-based EMP are that similarity information generated from the nonlinear interaction of different kernels is fully exploited, and the discriminability of the classes of interest is deeply enhanced. Experiments are conducted on three real hyperspectral data sets. The experimental results show that the proposed method achieves better performance for hyperspectral image classification, compared with several state-of-the-art algorithms. The MultiSE EMPs can provide much higher classification accuracy than using a single-SE EMP.

The large amount of spectral and spatial information contained in hyperspectral imagery has provided great opportunity to effectively characterize and identify the surface materials of interest. As a novel feature extraction technique, a series of Gabor wavelet filters with different scales and frequencies was applied on hyperspectral data to extract spectral-spatial-combined features, which produced impressive performance on pixel-oriented classification. However, the incredibly large number of Gabor features could cause too much burden for onboard computation, limiting the efficiency of the method. To make matters worse, due to the nonhomogeneous spatial distribution of materials as well as the different characteristics of the constructed Gabor filters, some Gabor features could have a smaller or even negative impact on material representation, deteriorating the classification accuracy eventually. In this paper, a Gabor cube selection based multitask joint sparse representation approach, abbreviated as GS-MTJSRC, was proposed for hyperspectral image classification. First, based on the Fisher discrimination criterion, the most representative Gabor cubes for each class were picked out. Next, under multitask joint sparse representation framework, a coefficient vector could be obtained for each test sample with the selected Gabor cube features, which could be directly used for the following residual-based classification. Experimental results on three real hyperspectral data sets with different characteristics and spatial resolutions demonstrated the feasibility and efficiency of the proposed method.

A novel hyperspectral image (HSI) classification method by the probabilistic fusion of pixel-level and superpixel-level classifiers is proposed. Generally, pixel-level classifiers based on spectral information only may generate “salt and pepper” result in the classification map since spatial correlation is not considered. By incorporating spatial information in homogeneous regions, the superpixel-level classifiers can effectively eliminate the noisy appearance. However, the classification accuracy will be deteriorated if undersegmentation cannot be fully avoided in superpixel-based approaches. Therefore, it is proposed to adaptively combine both the pixel-level and superpixel-level classifiers, to improve the classification performance in both homogenous and structural areas. In the proposed method, a support vector machine classifier is first applied to estimate the pixel-level class probabilities. Then, superpixel-level class probabilities are estimated based on a joint sparse representation. Finally, the two levels of class probabilities are adaptively combined in a maximum a posteriori estimation model, and the classification map is obtained by solving the maximum optimization problem. Experimental results on real HSI images demonstrate the superiority of the proposed method over several well-known classification approaches in terms of classification accuracy.

In recent years, many studies on hyperspectral image classification have shown that using multiple features can effectively improve the classification accuracy. As a very powerful means of learning, multiple kernel learning (MKL) can conveniently be embedded in a variety of characteristics. This paper proposes a class-specific sparse MKL (CS-SMKL) framework to improve the capability of hyperspectral image classification. In terms of the features, extended multiattribute profiles are adopted because it can effectively represent the spatial and spectral information of hyperspectral images. CS-SMKL classifies the hyperspectral images, simultaneously learns class-specific significant features, and selects class-specific weights. Using an $L_{1}$-norm constraint (i.e., group lasso) as the regularizer, we can enforce the sparsity at the group/feature level and automatically learn a compact feature set for the classification of any two classes. More precisely, our CS-SMKL determines the associated weights of optimal base kernels for any two classes and results in improved classification performances. The advantage of the proposed method is that only the features useful for the classification of any two classes can be retained, which leads to greatly enhanced discriminability. Experiments are conducted on three hyperspectral data sets. The experimental results show that the proposed method achieves better performances for hyperspectral image classification compared with several state-of-the-art algorithms, and the results confirm the capability of the method in selecting the useful features.

Cloud covers, which generally appear in optical remote sensing images, limit the use of collected images in many applications. It is known that removing these cloud effects is a necessary preprocessing step in remote sensing image analysis. In general, auxiliary images need to be used as the reference images to determine the true ground cover underneath cloud-contaminated areas. In this paper, a new cloud removal approach, which is called multitemporal dictionary learning (MDL), is proposed. Dictionaries of the cloudy areas (target data) and the cloud-free areas (reference data) are learned separately in the spectral domain. The removal process is conducted by combining coefficients from the reference image and the dictionary learned from the target image. This method could well recover the data contaminated by thin and thick clouds or cloud shadows. Our experimental results show that the MDL method is effective in removing clouds from both quantitative and qualitative viewpoints.

Cloud removal is an important goal for enhancing the utilization of optical remote sensing satellite images. Clouds dynamically affect the signal transmission due to their different shapes, heights, and distribution. In the case of thick opaque clouds, pixel replacement has been commonly adopted. For thin clouds, pixel correction techniques allow the effects of thin clouds to be removed while retaining the remaining information in the contaminated pixels. In this paper, we develop a new method based on signal transmission and spectral mixture analysis for pixel correction which makes use of a cloud removal model that considers not only the additive reflectance from the clouds but also the energy absorption when solar radiation passes through them. Data correction is achieved by subtracting the product of the cloud endmember signature and the cloud abundance and rescaling according to the cloud thickness. The proposed method has no requirement for meteorological data and does not rely on reference images. Our experimental results indicate that the proposed approach is able to perform effective removal of thin clouds in different scenarios.

Optical remote sensing images are often contaminated by the presence of clouds. The development of cloud effect removal techniques can maximize the usefulness of multispectral or hyperspectral images collected in the spectral range from visible to mid infrared. This paper presents a new data reconstruction technique, via dictionary learning and sparse representation, to remove the cloud effects. Dictionaries of the cloudy data (target data) and the cloud free data (reference data) are learned separately in the spectral domain, where each atom represents a fine ground cover component under the two imaging conditions. In this study, it is found that the sparse coefficients of the reference data are the true weightings of each atom, which can be used to replace the cloud affected coefficients to achieve data correction. Experiments were conducted using Landsat 8 OLI data sets downloaded from the USGS website. The testing results show that clouds of various thickness and cloud shadows can be removed effectively using the proposed method.

Conventional methods for mixed pixel analysis have their limitations in performance when the scenario is highly mixed without pure endmembers or only virtual endmembers can be generated. Moreover, theses approaches do not address the endmember variability. In this study, a multiple endmembers extraction algorithm based on Archetypal Analysis (AA) is proposed to solve the above problems. AA aims at finding distinct patterns in the data and thus, is suitable for endmember extraction. It can also generate vitual pure archetypes when no pure samples exist in the data. Kernel version of AA is investigated for multiple endmember extraction. Informative samples which contribute to the generation of each endmember class can be extracted and used as the multiple endmembers of a single ground cover type. Experimental results show that the multiple endmembers unmixing method using Kernal AA achieves more realistic unmixing results than single endmember based unmixing.

A novel multiple-kernel learning (MKL) model is proposed for urban classification to integrate heterogeneous features (HF-MKL) from two data sources, i.e., spectral images and LiDAR data. The features include spectral, spatial, and elevation attributes of urban objects from the two data sources. With these heterogeneous features (HFs), the new MKL model is designed to carry out feature fusion that is embedded in classification. First, Gaussian kernels with different bandwidths are used to measure the similarity of samples on each feature at different scales. Then, these multiscale kernels with different features are integrated using a linear combination. In the combination, the weights of the kernels with different features are determined by finding a projection based on the maximum variance. This way, the discriminative ability of the HFs is exploited at different scales and is also integrated to generate an optimal combined kernel. Finally, the optimization of the conventional support vector machine with this kernel is performed to construct a more effective classifier. Experiments are conducted on two real data sets, and the experimental results show that the HF-MKL model achieves the best performance in terms of classification accuracies in integrating the HFs for classification when compared with several state-of-the-art algorithms.

Due to the limited spatial resolution of multispectral/hyperspectral data, mixed pixels widely exist and various spectral unmixing techniques have been developed for information extraction at the subpixel level in recent years. One of the challenging problems in spectral mixture analysis is how to model the data of a primary class. Given that the within-class spectral variability (WSV) is inevitable, it is more realistic to associate a group of representative spectra with a pure class. The unmixing method using the extended support vector machines (eSVMs) has handled this problem effectively. However, it has simplified WSV in the mixed cases. In this paper, a further development of eSVMs is presented to address two problems in multiple-endmember spectral mixture analysis: 1) one mixed pixel may be unmixed into different fractions (model overlap); and 2) one fraction may correspond to a group of mixed pixels (fraction overlap). Then, spectral unmixing resolution (SUR) is introduced to characterize how finely the mixture in a mixed pixel can be quantified. The quantitative relationship between SUR and WSV of endmembers is derived via a geometry analysis in support vector machine feature space. Thus, the possible SUR can be estimated when multiple endmembers for each class are given. Moreover, if the requirement of SUR is fixed, the acceptance level of WSV is then limited, which can be used as a guide to remove outliers and purify endmembers for each primary class. Experiments are presented to illustrate model and fraction overlap problems and the application of SUR in uncertainty analysis of spectral unmixing.

Multiple scattering within a mixed pixel results in a nonlinear effect on the measured spectra in remotely sensed imagery. This study provides a quantitative assessment of multiple scattering in the reflectance of semiarid shrublands and explores its relationship to the characteristics of shrubs (density and height) and imaging parameters (wavelength and viewing angles). Field measurements were conducted at the southern fringe of the Otindag Sandy Land in China. A Monte Carlo ray tracing model, the Forest LIGHT interaction model (FLIGHT), was applied to simulate the multiple scattering results. FLIGHT simulation results were first evaluated against field measurements and then compared with a Landsat-8 OLI image. Results show that: 1) the contribution of multiple scattering to the spectra of a scene increases linearly with the fractional cover of vegetation and crown height; 2) in general, multiple scattering has a stronger effect on the near-infrared (NIR) domain than on the visible bands; 3) shadows significantly strengthen the multiple scattering effect, specifically within the visible bands; and 4) 80 to 100% of the total multiple scattering is caused by the second-order scattering within the visible bands and 60% to 90% within the NIR band. This study helps to improve our understanding of the multiple scattering effect and to select between linear and nonlinear spectral unmixing models to solve the abundances of shrubs and soil in mixed pixels.

Object-oriented remote sensing image classification is becoming more and more popular because it can integrate spatial information from neighboring regions of different shapes and sizes into the classification procedure to improve the mapping accuracy. However, object identification itself is difficult and challenging. Superpixels, which are groups of spatially connected similar pixels, have the scale between the pixel level and the object level and can be generated from oversegmentation. In this paper, we establish a new classification framework using a superpixel-based graphical model. Superpixels instead of pixels are applied as the basic unit to the graphical model to capture the contextual information and the spatial dependence between the superpixels. The advantage of this treatment is that it makes the classification less sensitive to noise and segmentation scale. The contribution of this paper is the application of a graphical model to remote sensing image semantic segmentation. It is threefold. 1) Gradient fusion is applied to multispectral images before the watershed segmentation algorithm is used for superpixel generation. 2) A probabilistic fusion method is designed to derive node potential in the superpixel-based graphical model to address the problem of insufficient training samples at the superpixel level. 3) A boundary penalty between the superpixels is introduced in the edge potential evaluation. Experiments on three real data sets were conducted. The results show that the proposed method performs better than the related state-of-the-art methods tested.

Automated cell segmentation for microscopy cell images has recently become an initial step for further image analysis in cell biology. However, microscopy cell images are easily degraded by noise during the readout procedure via optical-electronic imaging systems. Such noise degradations result in low signal-to-noise ratio (SNR) and poor image quality for cell identification. In order to improve SNR for subsequent segmentation and image-based quantitative analysis, the commonly used state-of-art restoration techniques are applied but few of them are suitable for corrupted microscopy cell images. In this paper, we propose a multistaged method based on a novel integration of trend surface analysis, quantile-quantile plot, bootstrapping, and the Gaussian spatial kernel for the restoration of noisy microscopy cell images. We show this multistaged approach achieves higher performance compared with other state-of-art restoration techniques in terms of peak signal-to-noise ratio and structure similarity in synthetic noise experiments. This paper also reports an experiment on real noisy microscopy data which demonstrated the advantages of the proposed restoration method for improving segmentation performance.

The presence of atmosphere can cause obstructions to satellite remote sensing by absorbing and scattering the electromagnetic energy. Therefore, transmittance of the atmosphere is an important factor to consider in a sensing system design1. Also the weather conditions such as the levels of the haze, dust or mist present in the environment, introduce distortion. Relative distributions of the brightness values of images can be different depending on the seasonal effect, termed radiometric inconsistency, which is solely dependent upon the solar radiation, illumination and reflectivity effects of the object and the conditions of the atmosphere during that time. Since they change frequently, multi-temporal data have low consistency over time. The inconsistency present in the remote sensed satellite images taken for sequential analysis can cause misguiding informaiton widely used in a range of oceanographic, terrestrial and atmospheric applications, such as land-cover mapping, environmental monitoring and disaster management. Degraded multi-temporal images needs to be checked and reconstructed before it can be used. In this paper a gradient adjusted temporal prediction approach has been used to predict or approximate the recent corrupted image using previous reference image.

The detection of cirrus cloud has historically been difficult due to the lack of a 1.375μm wavelength band in earlier Landsat ETM+ imagery. Landsat 8 OLI has addressed this problem by adding a new cirrus band at this wavelength. This paper presents a study of the effectiveness of utilizing the newly available data for this purpose. An image-based method is developed for cirrus cloud contamination correction. The relationship between a visible or infrared band and the cirrus band is estimated via a linear regression using the data in a homogenous land cover area. The key issue of how to automatic identify homogenous background from the cirrus contaminated data is addressed. The images corrected by our method show satisfactory quality.

Finding a subspace which consists of the most informative features for reliable hyperspectral image classification is a challenging task. Feature reduction is often achieved via feature selection and feature extraction techniques. In this letter, a hybrid approach which combines both treatments is proposed. Principal Component Analysis (PCA) is applied as a preprocessing step so that each of the new features is generated from the complete set of the original spectral bands. Feature selection is then performed effectively using a normalized Mutual Information (nMI) measure with two constraints to maximize general relevance and minimize redundancy in the selected subspace. The proposed algorithm (PCA-nMI) is tested on hyperspectral images and the experimental results show that the modifications give significant improvement in terms of classification accuracy.

While an increasing number of satellite images are collected over a regular period in order to provide regular spatiotemporal information on land-use and land-cover changes, there are very few compression schemes in remotely sensed imagery that use historical data as a reference. Just as individual images can be compressed for separate transmission by taking into account their inherent spatial and spectral redundancies, the temporal redundancy between images of the same scene can also be exploited for sequential transmission. In this letter, we propose a nonlinear elastic method based on the general relationship to predict adaptively the current image from a previous reference image without any loss of information. The main feature of the developed method is to find the best prediction for each pixel brightness value individually using its own conditional probabilities to the previous image, instead of applying a single linear or nonlinear model. A codebook is generated to record the nonlinear point-to-point relationship. This temporal lossless compression is incorporated with spatial- and spectral-domain predictions, and the performances are compared with those of the JPEG2000 standard. The experimental results show an improved performance by more than 5%.

The aim of this analysis is to develop a subspace detection technique using a hybrid approach which combines nonlinear feature extraction and feature selection for the task of hyperspectral image classification. In the proposed approach Kernel Principal Component Analysis (KPCA) is applied at the first step to generate the new features from the original data. Then pixel based spatial correlation is measured for each of the KPCA images to rank them based on their spatial objects/contents. These KPCA and spatial correlation based ranking scores are combined to obtain an informative subset of features. The experimental analysis conducted on a real hyperspectral image acquired by the AVIRIS sensor shows the advantage of the proposed approach in terms of classification accuracy.

The paper presents a supervised classification method based on superpixels and Markov random field (MRF). Hyperspectral image is over-segmented into superpixels that are as basic unit of Markov random field instead of operating at the pixel level. Adaptive weight coefficient is introduced to determine contextual relationship between superpixels. Support vector machines are implemented for better estimation of spectral contribution to this approach. An experiment of real hyperspectral image reveals efficient performance.

This letter presents a new technique for clustering hyperspectral images that exploits neighborhood-constrained spatial information. The main feature of the proposed method is the introduction of a neighborhood homogeneity index (NHI) and the use of this index to measure the spatial homogeneity in a local area. A new similarity measurement integrates NHI and spectral information using an adaptive distance norm for clustering. The performance of the proposed neighborhood-constrained-clustering algorithm was assessed through a synthetic image and a real hyperspectral image and compared with those obtained by advanced spectral-spatial clustering algorithms. Experimental results show that the proposed scheme gives better performances.

Wavelet packet analysis (WPA) and gray model (GM) are investigated for nonlinear unsupervised feature extraction of hyperspectral remote sensing data in this letter. Treated as derivative series, a hyperspectral response curve of each pixel is decomposed into an approximation and various detailed compositions by WPA, and then, GM is continuously applied to find the relationship among those detailed compositions. Cluster-space representation is used for determining the optimal wavelet. New extracted features can reveal the intrinsic identities of hyperspectral data. Experimental results show the feasibility and reliability of our proposed method in terms of classification accuracy.

Hyperspectral sensors record the reflectance from the Earth's surface over the full range of solar wavelengths with high spectral resolution. The resulting high-dimensional data contain rich information for a wide range of applications. However, for a specific application, not all the measurements are important and useful. The original feature space may not be the most effective space for representing the data. Feature mining, which includes feature generation, feature selection (FS), and feature extraction (FE), is a critical task for hyperspectral data classification. Significant research effort has focused on this issue since hyperspectral data became available in the late 1980s. The feature mining techniques which have been developed include supervised and unsupervised, parametric and nonparametric, linear and nonlinear methods, which all seek to identify the informative subspace. This paper provides an overview of both conventional and advanced feature reduction methods, with details on a few techniques that are commonly used for analysis of hyperspectral data. A general form that represents several linear and nonlinear FE methods is also presented. Experiments using two widely available hyperspectral data sets are included to illustrate selected FS and FE methods.

In this paper, we address a spectral unmixing problem for hyperspectral images by introducing multiple-kernel learning (MKL) coupled with support vector machines. To effectively solve issues of spectral unmixing, an MKL method is explored to build new boundaries and distances between classes in multiple-kernel Hilbert space (MKHS). Integrating reproducing kernel Hilbert spaces (RKHSs) spanned by a series of different basis kernels in MKHS is able to provide increased power in handling general nonlinear problems than traditional single-kernel learning in RKHS. The proposed method is developed to solve multiclass unmixing problems. To validate the proposed MKL-based algorithm, both synthetic data and real hyperspectral image data were used in our experiments. The experimental results demonstrate that the proposed algorithm has a strong ability to capture interclass spectral differences and improve unmixing accuracy, compared to the state-of-the-art algorithms tested.

The scale invariant feature transform (SIFT) is a widely used method for image registration and object recognition. The SIFT method is well known for its ability to identify objects at varying scales and rotations among clutter and occlusion with very fast processing time. The application of SIFT on multi-modal remote sensing images for image registration purposes, however, often results in inaccurate and sometimes incorrect matching. Commonly a very large number of feature points are generated from a remote sensing image but a very small number of feature points are matched giving a high false alarm rate. This paper proposes a method containing several modifications to improve the feature matching performance of the SIFT algorithm by adapting it to suit the characteristics of remote sensing images. The proposed method leads to more matching points with a significantly higher rate of correct matches.

Hyperspectral images contain a large amount of information which presents a major challenge for efficient classification. In this paper the information content of each spectral band is analyzed and an improved feature selection technique is proposed for the minimization of dependent information while maximizing the relevancy based on normalized mutual information (NMI). Experimental results are provided for comparisons among some relevant and recentmethods for hyperspectral feature selection in terms of their classification accuracy using real hyperspectral images.

Extended support vector machines (ESVM) was introduced recently for spectral unmixing. It models a class using a group of representative spectra to accommodate within class spectral variation. This paper presents a further geometry analysis of this method, and an improved ESVM is developed, which takes into account both within-class spectral variability and within each mixed case. The experiments illustrate that the new proposed algorithm can obtain more realistic unmixing results.

This paper investigates the usage of Johansen Cointegration Test for adaptive target detection with hyperspectral remote sensing data. Johansen Cointegration Test aims at mining long-term equilibrium relationship, which refers to the condition that if pairs of non-stationary series share similar tendencies, their linear combination could be stationary. Hyperspectral data are highly non-stationary series, but there should be similar patterns among the hyperspectral response curves of same materials. To be treated as derivative series, given hyperspectral response curves will be matched with the standard spectrum via Johansen Cointegration Test. The test statistics will be compared to a preset threshold to judge whether they are target or not. Quantitative experiments show that the proposed method performs better than a few other adaptive detection methods tested.

Categorization based on objects is an effective way to integrate spectral and spatial information into remote sensing image classification. In this paper, we establish a classification framework which represents objects by super pixels. The non-parametric k-NN approach is chosen for this super pixel based method, as it is simple and free of class data distribution. A new descriptor for the features distribution of each super pixel, called 4-D color histograms, is used for both spectral and texture information. This descriptor provides a better tolerance for value fluctuations inside the super pixel. Furthermore, the Ç2distance is used as the measure of the similarity between color histograms of the super pixels. Experiments are conducted to illustrate the application of the proposed method.

Hyperspectral image processing has attracted high attention in remote sensing fields. One of the main issues is to develop efficient methods for dimensionality reduction via feature extraction. This letter proposes a new nonlinear unsupervised feature extraction algorithm using Hurst and Lyapunov exponents to reveal local and general spectral profiles, respectively. A hyperspectral reflectance curve from each pixel is regarded as a time series, and it is represented by Hurst and Lyapunov exponents. These two new features are then used to overcome the Hughes problem for reliable classification. Experimental results show that the proposed method performs better than a few other feature extraction methods tested.

Conditional random fields (CRF) have been introduced to remote sensing image classification recently to integrate contextual information into remote sensing classification. It employs the spatial property on both pixel's spectral data and labels. However, this leads to a large number of model parameters to train. In this letter, the training efficiency is improved by modifying the conventional CRF model. At the same time, a class boundary constraint is imposed into this framework to avoid over correction. The advantages of the developed method are demonstrated in the experimental results using real remotely sensed data.

One of the most important steps in data fusion is image registration. Automatic image-to-image registration for images captured by different sensors traditionally requires the use of information-theoretic similarity measures such as mutual information. Recently, a new similarity measure known as cross-cumulative residual entropy (CCRE) has been proposed for multimodal image registration in medical imaging applications. In this paper, we investigate the use of CCRE for multisensor registration of remote sensing imagery. In particular, we investigate the extreme case of registering synthetic aperture radar images to optical images. We also propose a novel extension to the Parzen-window optimization approach proposed by Thévenaz which involves applying partial volume interpolation in the calculation of the gradients of the similarity measure. Our experimental results show that our proposed approach which uses CCRE as the similarity measure and partial volume interpolation in the optimization procedure provides superior performance to other approaches investigated.

Endmember extraction (EE) is the first step in hyperspectral data unmixing. N-FINDR is one of the most commonly used EE algorithms. Nevertheless, its computational complexity is high, particularly, for a large data set. Following a parallel version of N-FINDR, i.e., P-FINDR, further improvements are presented in this paper. First, generic endmember re-extraction operation (GERO) and multiple search paths are introduced such that multiple endmembers are extracted in parallel. Second, by making full use of the advantages of the proposed algorithms, two extended schemes, i.e., extended mapping rule and multiple-stage GERO are presented, which can reduce synchronous cost and provide steady parallel performance. In experiments, the proposed algorithms have been quantitatively evaluated. The results demonstrate that they can outperform the conventional parallel computing and do not degrade the quality of EE.

Finding the most informative features from high dimensional space for reliable class data modeling is one of the most challenging problems in hyperspectral image classification. The problem can be address using two basic techniques: feature selection and feature extraction. One of the most popular feature extraction methods is Principal Component Analysis (PCA), however its components are not always suitable for classification. In this paper, we present a feature reduction method (MI-PCA) which uses a nonparametric mutual information (MI) measure on the components obtained via PCA. Supervised classification results using a hyperspectral data set confirm that the new MI-PCA technique provides better classification accuracy by selecting more relevant features than when using either PCA or MI on the original data.

Spatial resolution is one of the central issues in land cover mapping from remote sensing imagery, and sub-pixel land cover mapping is difficult and challenging in this domain. Soft classification can provide more information than hard classification. However, the spatial location of land cover compositions within each pixel is unknown. To solve this problem, super-resolution mapping methods have been developed in recent years. In this paper, a landscape structure based approach for super-resolution land cover mapping is introduced to generate super-resolution land cover maps from remote sensing data. The method was used to map simulated target images and a real landscape and the results indicate that this approach has the ability to reconstruct complicated landscape with linear features, and large or small patches relative to the pixel size.

Feature extraction is an important research aspect for hyperspectral remote sensing image classification to reduce the complexity and improve the classification accuracy. In this paper, a new feature extraction method, Kernel based Local Fisher Discriminative Analysis (KLFDA), is applied to hyperspectral remote sensing processing. This method integrates the advantages of conventional supervised Fisher Discriminative Analysis and unsupervised Locality Preserving Projection methods. Several experiments using the real images have been conducted, which indicate a high efficiency of this algorithm for hyperspectral image classification.

An adaptive Markov random field (MRF) approach is proposed for classification of hyperspectral imagery in this letter. The main feature of the proposed method is the introduction of a relative homogeneity index for each pixel and the use of this index to determine an appropriate weighting coefficient for the spatial contribution in the MRF classification. In this way, overcorrection of spatially high variation areas can be avoided. Support vector machines are implemented for improved class modeling and better estimate of spectral contribution to this approach. Experimental results of a synthetic hyperspectral data set and a real hyperspectral image demonstrate that the proposed method works better on both homogeneous regions and class boundaries with improved classification accuracy.

In the past decades, spectral unmixing has been studied for deriving the fractions of spectrally pure materials in a mixed pixel. However, limited attention has been given to the collinearity problem in spectral mixture analysis. In this paper, quantitative analysis and detailed simulations are provided, which show that the high correlation between the endmembers, including the virtual endmembers introduced in a nonlinear model, has a strong impact on unmixing errors through inflating the Gaussian noise. While distinctive spectra with low correlations are often selected as true endmembers, the virtual endmembers formed by their product terms can be highly correlated. It is found that a virtual-endmember-based nonlinear model generally suffers more from collinearity problems compared to linear models and may not perform as expected when the Gaussian noise is high, despite its higher modeling power. Experiments were conducted on a set of in situ measured data, and the results show that the linear mixture model performs better in 61.5% of the cases.

The 19 papers in this special issue focus on the state-of-the-art and most recent developments in the area of spectral unmixing of remotely sensed data.

In this paper we address the problem of image fusion between imagery acquired by trichromatic sensors and hyperspectral imagers. We do this by presenting a method aimed at registering a high-resolution trichromatic image with lower resolution hyperspectral data. The method presented here maps the hyperspectral image into the grayscale image so as to employ the cross cumulative residual entropy for purposes of multimodal registration. We illustrate the utility of our approach by presenting registration results on a set of surveillance image pairs consisting of a set of high-oblique colour and hyperspectral images.

Multi-sensor image registration is a challenging task in remote sensing. Considering the fact that multi-sensor devices capture the images at different times, multi-spectral image registration is necessary for data fusion of the images. Several conventional methods for image registration suffer from poor performance due to their sensitivity to scale and intensity variation. The scale invariant feature transform (SIFT) is widely used for image registration and object recognition to address these problems. However, directly applying SIFT to remote sensing image registration often results in a very large number of feature points or keypoints but a small number of matching points with a high false alarm rate. We argue that this is due to the fact that spatial information is not considered during the SIFT-based matching process. This paper proposes a method to improve SIFT-based matching by taking advantage of neighborhood information. The proposed method generates more correct matching points as the relative structure in different remote sensing images are almost static.

This paper investigates the affect of variation of patterns in protein profiles to the identification of disease-specific biomarkers. A correlation-based cluster-space transform is applied to mass spectral data for predicting major adverse cardiac events (MACE). Training and testing data are transformed into cluster spaces by correlation distance based clustering, respectively. Data in the testing cluster that falls into a pair of training clusters is classified by a supervised classifier. Experiment results have shown that proteomic spectra of MACE which vary with certain patterns could be separated by the correlation-based clustering. The cluster-space transform allows better classification accuracy than single-clustered class method for separating disease and healthy samples.

In this paper, we propose a new super resolution (SR) method called the maximum a posteriori based on a universal Hidden Markov Tree (HMT) model for remote sensing images. The HMT theory is first used to set up a prior model for reconstructing super resolved images from a sequence of warped, blurred, subsampled, and noise-contaminated low-resolution (LR) images. Because the wavelet coefficients of images can be well characterized as a mixed Gaussian distribution, an HMT model is better able to capture the dependences between multiscale wavelet coefficients. The new method is tested first against simulated LR views from a single Landsat7 panchromatic scene and, then, with actual data from four Landsat7 panchromatic images captured on different dates. Both tests show that our method achieves better SR results both visually and quantitatively than other methods.

In this paper, a selective data compression scheme is developed to combine the need for efficient data transmission and the receivers' interest in changes presented in the data by taking the advantage of the fact that receivers hold a copy of previous data. Three-step pre-processing is introduced in this paper. Firstly we separate the unchanged areas (the majority) of the image from the changed areas between successive images of the same area. Secondly, the bands which are sensitive to the changes are identified with the aid of statistical measures. Finally, a binary index image of each band is generated to indicate the two categories. Following this pre-processing, compression of the unchanged areas and changed areas is conducted separately. In this way, different compression algorithms can be applied to each case. As the majority data will be unchanged and only a subset of bands reflects the changes, high compression rate is achievable.

In this letter, the application of superresolution (SR) techniques to multispectral image clustering and classification is investigated and tested using satellite data. A set of multispectral images with better spatial resolution is obtained after an SR technique is applied to several data sets recorded within a short period over a study area. Improved clustering and classification performance is demonstrated visually and quantitatively by comparison with the original low-resolution data or enlarged images using a conventional interpolation method. This letter illustrates the possibility and feasibility of the use of SR reconstruction for the classification of remote sensing data, which is encouraging as a means of breaking through current satellite detectors' resolution limits.

In this letter, the supervised classification algorithm support vector machines is extended to map both pure pixels and mixed pixels using hyperspectral data. The margins between the hyperplanes formed by the pixels on the class boundaries are recognized as mixed region, and the space beyond this region is related to pure pixels. In this way, each endmember is modeled by a set of training samples instead of a single (representative) spectrum to accommodate the variations within the relative pure pixels due to system noise. Unmixing outputs generate an integrated soft- and hard-classification map. The better performance comparing with conventional spectral unmixing method was demonstrated using hyperspectral data sets.

Several spectral unmixing techniques have been developed for subpixel mapping using hyperspectral data in the past two decades, among which the fully constrained least squares method based on the linear spectral mixture model (LSMM) has been widely accepted. However, the shortage of this method is that the Euclidean spectral distance measure is used, and therefore, it is sensitive to the magnitude of the spectra. While other spectral matching criteria are available, such as spectral angle mapping (SAM) and spectral information divergence (SID), the current unmixing algorithm is unable to be extended to these measures. In this paper, we propose a unified subpixel mapping framework that models the unmixing process as a best match of the unknown pixel's spectrum to a weighted sum of the endmembers' spectra. We introduce sequential quadratic programming to solve the nonlinear optimization problem encountered in the implementation of this framework. The main feature of this proposed method is that it is not restricted to any particular similarity measures. Experiments were conducted with both simulated and Hyperion data. The tests demonstrated the proposed framework's advantage in accommodating various spectral similarity measures and provided performance comparisons of the Euclidean distance measure with other spectral matching criteria including SAM, spectral correlation measure, and SID.

In this paper, we propose a new super resolution method Maximum a Posteriori based on a universal hidden Markov tree model (MAP-uHMT) for remote sensing images. The hidden Markov tree theory in the wavelet domain is used to set up a prior model for reconstructing super resolution images from a sequence of warped, blurred, sub-sampled and contaminated low resolution images. Both the simulation results with a Landsat7 panchromatic image and actual results with four Landsat7 panchromatic images which were captured on different dates show that our method achieves better super resolution images both visually and quantitatively than other methods, based on PSNR in the simulation and derived PSF with actual data.

The accuracy of Land Use/Land Cover data derived from remote sensing images is critical for many applications. Classification error is caused by the interaction of numerous factors, including landscape characteristics, sensor resolution, preprocessing algorithms, and classification procedures. The purpose of this paper is to extend the evaluation of the Collection 4 MODIS land cover product (MOD12Q1) to Shandong Province, east China, and to analyze the distribution of errors from a landscape pattern perspective. Logistic regression was employed to assess the impact of landscape characteristics on classification accuracy. A correct classification probability map based on logistic model was given for describing MOD12 land cover product's error distribution. The results indicate that classification accuracy increases as land cover patch size increases and as heterogeneity decreases.

A straightforward method is presented for determining the most appropriate weighting of the spectral and spatial contributions in the Markov random field approach to context classification. The spectral and spatial components are each normalized to fall in the range (0,1) after which the appropriate value for the weighting coefficient can determined simply, guided by an assessment of the importance of the spatial contribution. Experimental results are presented using an artificial data set and real data recorded by the Landsat Thematic Mapper and Airborne Visible/Infrared Imaging Spectrometer.

A method is presented for supplementing the training set in maximum likelihood classification of hyperspectral data to mitigate the Hughes phenomenon. Based on the idea that the near neighbors of training pixels are likely to come from the same class, measures are proposed to assess neighbors as potential candidates so that those selected give improved class statistics and classification accuracy.

A simple method is presented for determining the most appropriate weighting of the spectral and spatial contributions in the Markov random field approach to context classification. The spectral and spatial components are each normalized to fall in the range (0,1) after which the appropriate value for the weighting coefficient is determined. A simple, systematic evaluation of the impact of the choice of weight on the ultimate classification results achieved is possible as a methodology for optimum weight selection. Experimental results are presented using an artificial data set and real data recorded by the TM and AVIRIS sensors.

In this letter, a geometry-based feature-selection method is proposed for efficient analysis of hyperspectral imagery. It searches the vertices that form the largest simplex iteratively in pixel space. These vertices are representative subsets of spectral bands. A distance measure is introduced in the simplex volume comparison for fast implementation of the proposed method. Fast principal component analysis and spectral band indexing are suggested for data preprocessing. This method can be implemented in supervised or unsupervised manner. It is automatic, fast, and distribution-free. Experimental results show the superiority of the proposed method in terms of quality and speed

A relaxation scheme is proposed in which Dempster-Shafer evidential theory is used to bring the effect of the spatial neighborhood of a pixel into a classification. The benefits include the ability to incorporate uncertainty in the neighborhood information, allowing a stopping criterion to be devised based on increasing the uncertainty contribution of the neighborhood to unity within a prescribed number of iterations. The number of iterations to be used is governed by several factors, including an estimate of how far out in the neighborhood pixels are assumed to be influential. As with standard relaxation labeling, but unlike many other context-sensitive methods, the evidential approach can be initialized from the results of a separate point statistical classification of the image; it is also consistent with multisource analyses based on evidential methods for fusion. A variation of evidential relaxation using considerably simplified neighborhood information is also developed, illustrating that very good results can be obtained without detailed knowledge of the spatial properties of a scene. The new procedures are compared experimentally with standard probabilistic relaxation and the application of Markov random fields

A fast k-nearest neighbor algorithm is presented which combines k-NN with a cluster-space data representation. Implementation of the algorithm is easier, and classification time can be significantly reduced. Computer-generated data show the modified k-NN retains the advantage of nonparametric analysis but with significant reduction in computational load. Results from tests carried out with Hyperion data demonstrate that the simplification has little effect on classification performance, and yet efficiency is greatly improved.

A relaxation scheme is proposed in which Dempster- Shafer evidential theory is used to bring the effect of the spatial neighbourhood of a pixel into a classification. The benefits include the ability to incorporate uncertainty, both in the central pixel's labels and in those of its neighbours. A stopping criterion can be devised by increasing the uncertainty contribution of the neighbourhood to unity within a prescribed number of iterations. The number of iterations to be used is governed by several factors, including an estimate of how far out in the neighbourhood pixels are assumed to be influential. As with standard relaxation labelling, but unlike many other context sensitive methods, the evidential approach can be initialised from the results of a separate point statistical classification of the image.

Images compressed by current lossy compression techniques suffer from distortion that is uniformly distributed spatially and spectrally. We demonstrate that classification accuracy changes as a function of pixel class and spectral location (spectral bands) of the distortion and that the uniform distribution of distortion is therefore not an optimal approach to controlling distortion within an image. We show that a superior approach involves locating areas within the image whose classification accuracies are relatively insensitive to distortion and limiting the application of distortion to these areas. By following this approach, we show that substantial levels of compression-induced distortion can be tolerated without a significant reduction in subsequent classification accuracy. Hyperspectral images are prime candidates for data compression due to their inherent size. Lossy compression algorithms are attractive as they typically provide the most impressive levels of compression (compression ratio) but result in some distortion of the original image. The acceptability of this distortion depends on the end use of the data set that, in the case of hyperspectral images, invariably involves the use of computer-based tools to process the images into a set of classes representing the ground coverage or conditions present in the data. Compression-induced distortion tends to reduce the accuracy associated with the classification process, but the relationship between distortion and classification accuracy varies across different classes of data within the same image and is not particularly predictable. We demonstrate an alternative to the uniform application of distortion during compression that aims to locate spectral and spatial areas within an image where sensitivity to distortion is likely to be reduced. We then restrict the application of the compression-induced distortion to these areas of low sensitivity and show that the subsequent classification accuracies are superior to uniformly distorted images.

An extension of a newly developed cluster-space representation is applied to efficient data transmission and classification. Cluster-space classification, which is an automatic hybrid supervised and unsupervised classification procedure, can be performed in two stages. A "semiproduct" with low entropy is generated at the sender end. It is then transmitted to a range of users for further classification. Experiments using a HyMap dataset demonstrate the advantages in data transmission and the satisfactory classification accuracy.

Airborne HYMAP data were used to estimate the sideroxylonal-A concentration from the canopy reflectance of Eucalyptus melliodora. Modified partial least squares (MPLS) was applied to the extracted mean and maximum canopy spectra to estimate the sideroxylonal-A concentrations. The resultant coefficients of determination (R/sup 2/) are high (e.g. 0.89 in one case), and the identified absorption features (wavelengths) are closely consistent with those identified in a previous study in which near IR spectroscopy was used to measure the sideroxylonal-A concentration of ground freeze-dried leaves, and a spectroradiometer was used to estimate the sideroxylonal-A concentration of whole oven-dried leaves and fresh whole leaves.

In this paper, a simplified maximum likelihood classification method is applied to cluster-space hyperspectral data analysis. A principal components (PC) transformation is firstly used for data de-correlation, followed by cluster-space representation where information classes are associated with spectral clusters automatically. The simplified maximum likelihood classification treats the transformed data independent of the PC features, allowing the second-degree statistics of each cluster to be taken into account with reduced requirement on the number of training samples. Pixel labelling is undertaken by a combined decision based on its membership of belonging to defined clusters and the clusters' membership of belonging to information classes.

This paper presents a generalization of the hybrid supervised-unsupervised approach to image classification, and an automatic procedure for implementing it with hyperspectral data. Cluster-space representation is introduced in which clustered training data is displayed in a one-dimensional (1-D) cluster-space showing its probability distribution. This representation leads to automatic association of spectral clusters with information classes and the development of a cluster-space classification (CSC). Pixel labeling is undertaken by a combined decision based on its membership of belonging to defined clusters and the clusters' membership of belonging to information classes. The method provides a means of class data separability inspection, visually and quantitatively, regardless of the number of spectral bands used. The class modeling requires only that first degree statistics be estimated; therefore, the number of training samples required can be many fewer than when using Gaussian maximum likelihood (GML) classification. Experiments are presented based on computer generated data and AVIRIS data. The advantages of the method are demonstrated showing improved capacity for data classification.

A segmented, and possibly multistage, principal components transformation (PCT) is proposed for efficient hyperspectral remote-sensing image classification and display. The scheme requires, initially, partitioning the complete set of bands into several highly correlated subgroups. After separate transformation of each subgroup, the single-band separabilities are used as a guide to carry out feature selection. The selected features can then be transformed again to achieve a satisfactory data reduction ratio and generate the three most significant components for color display. The scheme reduces the computational load significantly for feature extraction, compared with the conventional PCT. A reduced number of features will also accelerate the maximum likelihood classification process significantly, and the process will not suffer the limitations encountered by trying to use the full set of hyperspectral data when training samples are limited. Encouraging results have been obtained in terms of classification accuracy, speed, and quality of color image display using two airborne visible/infrared imaging spectrometer (AVIRIS) data sets.

A simplified maximum likelihood classification technique for handling remotely sensed image data is proposed which reduces, significantly, the processing time associated with traditional maximum likelihood classification when applied to imaging spectrometer data, and copes with the training of geographically small classes. Several wavelength subgroups are formed from the complete set of spectral bands in the data, based on properties of the global correlation among the bands. Discriminant values are computed for each subgroup separately and the sum of discriminants is used for pixel labeling. Several subgrouping methods are investigated and the results show that a compromise among classification accuracy, processing time, and available training pixels can be achieved by using appropriate subgroup sizes.>