In real hyperspectral scenes, heterogeneous spatial details and noises make a single subspace assumptions unrealistic. In this letter, a novel transferable multiple tensor subspace learning scheme is proposed for super-resolution enhancement of hyperspectral image (HSI). The intrinsic assumption is that the nonlocal patch tensors extracted from HSIs are derived from multiple tensor low-rank subspaces, which is compatible with practical data distribution and may better characterize the complex structures underlying HSIs. The transferable subspace structures are embedded into both nonblind and semi-blind HSI super-resolution. The alternating direction method of multipliers (ADMMs) algorithm is derived for model learning. The superiority of our method is demonstrated by comprehensive experiments on both synthetic and real datasets.

High-dimensional image representation is a challenging task since data has the intrinsic low-dimensional and shift-invariant characteristics. Currently, popular methods, such as tensor-Singular Value Decomposition (t-SVD), have limited ability in expressing shift-invariant subspace knowledge underlying data. To these problem, we propose a high-dimensional image representation framework based on Tensor Convolution-like Low-Rank Dictionary (TCLRD), which considers the shift-invariant low-dimensional structure of a tensor-valued data by convolution-like low-rank dictionary learning and coefficient coding, to promote the high-dimensional image representation ability. To be specific, we first define the TCLRD framework with low-rank constraint for dictionary and coefficient, in which tensor factorization and tensor-tensor product over frequency domain can be understood as convolution-like operation when describing shift-invariant. Then, the tensor Schatten-p norm is introduced to verify that TCLRD has rational mathematical interpretation. We study the TCLRD minimization problem in tensor completion with the ADMM-based optimization algorithm. The efficient solving scheme with TCLRD is extendable to various low-rank models like tensor robust principal component analysis and subspace clustering, and prove their theoretical guarantees based on generalization error. Extensive experimental results demonstrate the proposed TCLRD methods are beyond state-of-the-arts in typical tasks, including image denoising, HSI completion and image clustering.

Hyperspectral image (HSI) exhibits high-quality spectral signals that convey subtle differences, enabling the discrimination of similar materials and providing a unique advantage for anomaly detection (AD). Fine spectral of anomalies can be effectively identified amidst heterogeneous background pixels. Given the similarity of materials in spatial and spectral dimensions, joint utilization of spatial and spectral information enhances detection performance. However, many existing AD approaches for HSIs usually achieve high accuracy at the expense of high computational complexity. In response to the requirements of practical detection scenarios-efficiency, robustness, and accuracy-this article introduces a rapid and robust AD algorithm through discriminative band selection for HSIs. We propose a spatial-spectral feature extraction strategy to ensure detection accuracy. Initially, to effectively mine context information across a broad spectral range, the HSI cube in space is partitioned into several groups using a coarse-to-fine strategy. Subsequently, we identify the most relevant and informative bands based on spatial local density and spectral information entropy, forming the coarse HSI bands subset. Following this, we design a multiband target-background ratio (MBTBR) to capture strongly discriminative bands, resulting in the fine HSI bands subset. Finally, we present an adaptively spatial—spectral feature extraction strategy to detect anomalous targets. Extensive experimental results on real hyperspectral datasets demonstrate that the proposed method achieves satisfactory performance compared to the state-of-the-art algorithms, validating its strong robustness and low computational complexity simultaneously.

Fusing low-resolution (LR) hyperspectral images (HSIs) with high-resolution (HR) multispectral images (MSIs) is a significant technology to enhance the resolution of HSIs. Despite the encouraging results from deep learning (DL) in HSI-MSI fusion, there are still some issues. First, the HSI is a multidimensional signal, and the representability of current DL networks for multidimensional features has not been thoroughly investigated. Second, most DL HSI-MSI fusion networks need HR HSI ground truth for training, but it is often unavailable in reality. In this study, we integrate tensor theory with DL and propose an unsupervised deep tensor network (UDTN) for HSI-MSI fusion. We first propose a tensor filtering layer prototype and further build a coupled tensor filtering module. It jointly represents the LR HSI and HR MSI as several features revealing the principal components of spectral and spatial modes and a sharing code tensor describing the interaction among different modes. Specifically, the features on different modes are represented by the learnable filters of tensor filtering layers, the sharing code tensor is learned by a projection module, in which a co-attention is proposed to encode the LR HSI and HR MSI and then project them onto the sharing code tensor. The coupled tensor filtering module and projection module are jointly trained from the LR HSI and HR MSI in an unsupervised and end-to-end way. The latent HR HSI is inferred with the sharing code tensor, the features on spatial modes of HR MSIs, and the spectral mode of LR HSIs. Experiments on simulated and real remote-sensing datasets demonstrate the effectiveness of the proposed method.

Due to the influence of imaging equipment or environment, a hyperspectral image (HSI) is often unavoidably blurred in the acquisition process, which results in the spatial and spectral information loss of the HSI. The existing HSI deblurring methods can address the problem, however, they neglect the intrinsic structured sparsity and thus reduce the deblurring performance. Aiming at this issue, we propose a new HSI deblurring method based on transformed structured sparsity with smoothness (TSSS). We first use the local piecewise smoothness to obtain the spatial and spectral sparsity of an HSI in the gradient domain. Then, to capture the refined sparsity, we exploit the transform sparsity learning framework to encode the structured sparsity self-adaptively in transform space, where the sparse structures of transformed operators can be depicted by Laplacian scale mixture (LSM), i.e., the sparsity can be expressed as the product of a hidden positive scalar multiplier and a Laplacian vector. The visual and quantitative comparisons of experimental results on three HSI datasets indicate that our method outperforms state-of-the-arts.

The fusion of infrared and visible images combines the information from two complementary imaging modalities for various computer vision tasks. Many existing techniques, however, fail to maintain a uniform overall style and keep salient details of individual modalities simultaneously. This paper presents an end-to-end Laplacian Pyramid Fusion Network with hierarchical guidance (HG-LPFN) that takes advantage of pixel-level saliency reservation of Laplacian Pyramid and global optimization capability of deep learning. The proposed scheme generates hierarchical saliency maps through Laplacian Pyramid decomposition and modal difference calculation. In the pyramid fusion mode, all sub-networks are connected in a bottom-up manner. The sub-network for low-frequency fusion focuses on extracting universal features to produce an opposite style while sub-networks for high-frequency fusion determine how much the details of each modality will be retained. Taking the style, details, and background into consideration, we design a set of novel loss functions to supervise both low-frequency images and full-resolution images under the guidance of saliency maps. Experimental results on public datasets demonstrate that the proposed HG-LPFN outperforms the state-of-the-art image fusion techniques.

Extensive works have been reported in hyperspectral images (HSIs) and multispectral images (MSIs) fusion to raise the spatial resolution of HSIs. However, the limited acquisition of HSIs has been an obstacle to such approaches. Spectral super-resolution (SSR) of MSI is a challenging and less investigated topic, which can also provide high-resolution synthetic HSIs. To deal with this high ill-posedness problem, we perform super-resolution enhancement of MSIs in the spectral domain by incorporating a spectral library as a priori. First, an aligned spectral library, which maps the open-source spectral library to a specific spectral library created for the reconstructed HR HSI, is represented. An intermediate latent HSI is obtained by fusing the spatial information from MSI and the hyperspectral information from a specific spectral library. Then, we use low-rank attribute embedding to transfer latent HSI into a robust subspace. Finally, a low-rank HSI dictionary representing the hyperspectral information is learned from the latent HSI. The adaptive sparse coefficient of MSI is obtained with a nonnegative constraint. By fusing these two terms, we get the final HR HSI. The proposed SSR model does not require any pretraining stages. We confirm the validity and superiority of our proposed SSR algorithm by comparing it with several benchmark state-of-the-art approaches on different datasets.

Recently, tensor sparsity modeling has achieved great success in the tensor completion (TC) problem. In real applications, the sparsity of a tensor can be rationally measured by low-rank tensor decomposition. However, existing methods either suffer from limited modeling power in estimating accurate rank or have difficulty in depicting hierarchical structure underlying such data ensembles. To address these issues, we propose a parametric tensor sparsity measure model, which encodes the sparsity for a general tensor by Laplacian scale mixture (LSM) modeling based on three-layer transform (TLT) for factor subspace prior with Tucker decomposition. Specifically, the sparsity of a tensor is first transformed into factor subspace, and then factor sparsity in the gradient domain is used to express the local similarity in within-mode. To further refine the sparsity, we adopt LSM by the transform learning scheme to self-adaptively depict deeper layer structured sparsity, in which the transformed sparse matrices in the sense of a statistical model can be modeled as the product of a Laplacian vector and a hidden positive scalar multiplier. We call the method as parametric tensor sparsity delivered by LSM-TLT. By a progressive transformation operator, we formulate the LSM-TLT model and use it to address the TC problem, and then the alternating direction method of multipliers-based optimization algorithm is designed to solve the problem. The experimental results on RGB images, hyperspectral images (HSIs), and videos demonstrate the proposed method outperforms state of the arts.

Hyperspectral–multispectral image (HSI-MSI) fusion relies on a robust degradation model and data prior, where the former describes the degeneration of HSI in the spectral and spatial domains, and the latter reveals the latent statistics of the expected high-resolution (HR) HSI. In practice, the degradation model is often unknown, and the data prior is usually too complicated to be expressed analytically. In this study, we propose a variational network for HSI-MSI fusion (VaFuNet), in which the degradation model and data prior are implicitly represented by a deep learning network and jointly learned from the training data. A variational fusion model regularized by deep prior is first proposed, and then, it is optimized via a half-quadratic splitting and unfolded into a deep network. The deep prior is implicitly represented by a proximity operator. Due to the structural self-similarity, HSI possesses structural recurrences across different scales. To exploit such nonlocal prior and enhance the representability of network, we also propose a multiscale nonlocal attention and embed it into the deep prior proximity. The degradation model and deep prior proximity are jointly learned via end-to-end training. Experimental results on simulated and real-life HSI datasets demonstrate the effectiveness of the proposed VaFuNet HSI-MSI fusion method.

Existing methods for tensor completion (TC) have limited ability for characterizing low-rank (LR) structures. To depict the complex hierarchical knowledge with implicit sparsity attributes hidden in a tensor, we propose a new multilayer sparsity-based tensor decomposition (MLSTD) for the low-rank tensor completion (LRTC). The method encodes the structured sparsity of a tensor by the multiple-layer representation. Specifically, we use the CANDECOMP/PARAFAC (CP) model to decompose a tensor into an ensemble of the sum of rank-1 tensors, and the number of rank-1 components is easily interpreted as the first-layer sparsity measure. Presumably, the factor matrices are smooth since local piecewise property exists in within-mode correlation. In subspace, the local smoothness can be regarded as the second-layer sparsity. To describe the refined structures of factor/subspace sparsity, we introduce a new sparsity insight of subspace smoothness: a self-adaptive low-rank matrix factorization (LRMF) scheme, called the third-layer sparsity. By the progressive description of the sparsity structure, we formulate an MLSTD model and embed it into the LRTC problem. Then, an effective alternating direction method of multipliers (ADMM) algorithm is designed for the MLSTD minimization problem. Various experiments in RGB images, hyperspectral images (HSIs), and videos substantiate that the proposed LRTC methods are superior to state-of-the-art methods.

A deep convolutional neural network (CNN) has shown its great potential in hyperspectral image (HSI) super-resolution (SR). Integrating CNN with attention mechanism is expected to boost the SR performance. However, how to learn attention along the spectral, spatial, and channel dimensions of HSI is still an open issue, and the current attention mechanism is not efficient in capturing long-range interdependency in HSI. In this letter, we first design a local 3-D attention module to learn the spectral-spatial-channel attention by exploiting local contextual information in HSI. Then, we propose a nonlocal 3-D attention module, in which the long-range interdependency in HSI can be exploited for attention learning. By jointly embedding the local and nonlocal attention in a residual 3-D CNN, a hybrid local and nonlocal 3-D attentive CNN can be built for HSI SR. The experimental results show that local and nonlocal attention formulation leads to competitive SR performance.

We propose a novel graph Laplacian-guided coupled tensor decomposition (gLGCTD) model for fusion of hyperspectral image (HSI) and multispectral image (MSI) for spatial and spectral resolution enhancements. The coupled Tucker decomposition is employed to capture the global interdependencies across the different modes to fully exploit the intrinsic global spatial-spectral information. To preserve local characteristics, the complementary submanifold structures embedded in high-resolution (HR)-HSI are encoded by the graph Laplacian regularizations. The global spatial-spectral information captured by the coupled Tucker decomposition and the local submanifold structures are incorporated into a unified framework. The gLGCTD fusion framework is solved by a hybrid framework between the proximal alternating optimization (PAO) and the alternating direction method of multipliers (ADMM). Experimental results on both synthetic and real data sets demonstrate that the gLGCTD fusion method is superior to state-of-the-art fusion methods with a more accurate reconstruction of the HR-HSI.

Various existing anomaly detection (AD) technologies focus on the background feature extraction and suppression, which serves as a crucial step to extrude anomalies from the hyperspectral imagery (HSI). In this article, motivated by the advantages of the joint sparse representation (JSR) model for adaptive background base selection, a robust background feature extraction method through homogeneous region-based JSR is proposed and used for AD. By segmenting the scene from the spatial domain through an eight-connected region division operation based on the clustering result, a series of nonoverlapping homogeneous regions each sharing a common sparsity pattern are obtained. After discarding small regions, JSR is performed on each region with the dictionary constituted by the overall spectral items in the corresponding cluster. By calculating the usage frequency of dictionary atoms, the most representative background bases describing each background cluster are adaptively selected and then combined into the global background bases. In addition, considering the interference of noise on detection accuracy, an energy deviation-based noise estimation strategy is presented by analyzing the residual obtained from JSR. Finally, the anomaly response of each pixel is measured by comparing its projection energy obtained from the background orthogonal subspace projection with the noise energy in its corresponding region. The proposed method overcomes the shortcomings of traditional neighborhood-based JSR in the common sparsity pattern and anomaly proportion. The spatial characteristics of HSI are fully explored. Furthermore, the interference of noise on detection accuracy is eliminated. Experiments on four HSI data sets demonstrate the superiority of the proposed method.

Hyperspectral image super-resolution by fusing high-resolution multispectral image (HR-MSI) and low-resolution hyperspectral image (LR-HSI) aims at reconstructing high resolution spatial-spectral information of the scene. Existing methods mostly based on spectral unmixing and sparse representation are often developed from a low-level vision task perspective, they cannot sufficiently make use of the spatial and spectral priors available from higher-level analysis. To this issue, this paper proposes a novel HSI super-resolution method that fully considers the spatial/spectral subspace low-rank relationships between available HR-MSI/LR-HSI and latent HSI. Specifically, it relies on a new subspace clustering method named "structured sparse low-rank representation" (SSLRR), to represent the data samples as linear combinations of the bases in a given dictionary, where the sparse structure is induced by low-rank factorization for the affinity matrix. Then we exploit the proposed SSLRR model to learn the SSLRR along spatial/spectral domain from the MSI/HSI inputs. By using the learned spatial and spectral low-rank structures, we formulate the proposed HSI super-resolution model as a variational optimization problem, which can be readily solved by the ADMM algorithm. Compared with state-of-the-art hyperspectral super-resolution methods, the proposed method shows better performance on three benchmark datasets in terms of both visual and quantitative evaluation.

In this article, we proposed a thick cloud removal method for remote-sensing imagery based on multisource estimation. A convolutional-mapping-deconvolutional (CMD) network is proposed to estimate the cloud-free image directly from multisource reference images. Synthetic aperture radar (SAR) image and low-resolution heterogeneous (LRH) image, namely, image from a different optical sensor with lower spatial resolution, are used as reference images to recover the missing information in the cloud-contaminated high-resolution (HR) image. The CMD net is composed of three functional components: the convolutional layers for encoding, the mapping layer for feature transferring, and the deconvolutional layers for decoding. In the training procedure, HR images from cloud-free regions and their corresponding LRH and SAR reference images are used to train the CMD net. When the CMD net is fully trained, it is able to estimate the HR images with their corresponding LRH and SAR reference images. The LRH and SAR reference images are first encoded by the convolutional layers before being transferred to the feature space at HR by the mapping layer. The transferred features are then decoded into cloud-free HR image by the deconvolutional layers. Cloud-free regions in the cloud-contaminated HR image are used to further improve the estimated image via intensity normalization. At last, the cloudy pixels are replaced by their corresponding pixels from the estimated cloud-free HR image. Comparisons with several recently proposed multisource cloud removal methods show that our proposed method is superior as validated by quantitative indexes and visual inspections.

Convolutional neural networks (CNNs) are of great interest and have demonstrated remarkable performance in hyperspectral images (HSIs) classification. However, due to the current configuration of the convolution layers with a fixed kernel shape, regular CNNs are inherently limited in modeling the diverse land-cover structures, particularly in the cross-classes edge regions, where irregular class boundaries would lead to high classification errors. To address this issue, we propose a content-guided CNN (CGCNN) for HSI classification. Compared with the shape-fixed kernel in the traditional CNN, the proposed content-guided convolution adaptively adjusts its kernel shape according to the spatial distribution of land covers. The content pattern is reflected by a latent guide map automatically learned from HSI. Such content-adaptive kernel with CGCNN could suppress the irregularity and unexpected features in class boundaries and, thus, improve the feature learning in cross-classes regions. Based on the content-guided convolution, a novel guided feature extraction unit (GFEU) is constructed for spectral-spatial feature learning of HSI. Finally, the CGCNN classification framework is established by stacking multiple GFEUs with dense connection, which is helpful for mitigating the gradient vanishing and increasing the robustness to overfitting. Extensive experiments on several HSIs demonstrate that the proposed approach possesses great details' preserving ability and its performance outperforms other state-of-the-art methods.

Hyperspectral (HS) imaging has shown its superiority in many real applications. However, it is usually difficult to obtain high-resolution (HR) HS images through existing imaging techniques due to the hardware limitations. To improve the spatial resolution of HS images, this article proposes an effective HS-multispectral (HS-MS) image fusion method by combining the ideas of nonlocal low-rank tensor modeling and spectral unmixing. To be more precise, instead of unfolding the HS image into a matrix as done in the literature, we directly represent it as a tensor, then a designed nonlocal Tucker decomposition is used to model its underlying spatial-spectral correlation and the spatial self-similarity. The MS image serves mainly as a data constraint to maintain spatial consistency. To further reduce the spectral distortions in spatial enhancement, endmembers, and abundances from the spectral are used for spectral regularization. An efficient algorithm based on the alternating direction method of multipliers (ADMM) is developed to solve the resulting model. Extensive experiments on four HS image data sets demonstrate the superiority of the proposed method over several state-of-the-art HS-MS image fusion methods.

Conventional tensor completion (TC) methods generally assume that the sparsity of tensor-valued data lies in the global subspace. The so-called global sparsity prior is measured by the tensor nuclear norm. Such assumption is not reliable in recovering low-rank (LR) tensor data, especially when considerable elements of data are missing. To mitigate this weakness, this article presents an enhanced sparsity prior model for LRTC using both local and global sparsity information in a latent LR tensor. In specific, we adopt a doubly weighted strategy for nuclear norm along each mode to characterize global sparsity prior of tensor. Different from traditional tensor-based local sparsity description, the proposed factor gradient sparsity prior in the Tucker decomposition model describes the underlying subspace local smoothness in real-world tensor objects, which simultaneously characterizes local piecewise structure over all dimensions. Moreover, there is no need to minimize the rank of a tensor for the proposed local sparsity prior. Extensive experiments on synthetic data, real-world hyperspectral images, and face modeling data demonstrate that the proposed model outperforms state-of-the-art techniques in terms of prediction capability and efficiency.

Thin cloud removal is very important for optical remote sensing imagery. Different from thick cloud removal, the pixels contaminated by thin cloud still preserve some surface information. Therefore, thin cloud removal methods usually focus on suppressing the cloud influence instead of replacing the cloudy pixels. In this paper, we proposed a deep residual symmetrical concatenation network (RSC-Net) to make end-to-end cloud removal. The RSC-Net is based on an encoding-decoding framework consisting of multiple residual convolutional layers and residual deconvolutional layers. The feature maps of each convolutional layer are copied and concatenated to their symmetrical deconvolutional layers. Using supervised training with real Landsat-8 data, input samples include one cloudy image and one cloud-free reference image, and the cloud-free reference image also serves as the target. When the RSC-Net is fully trained, it is able to take cloudy image as input and produce cloud-free image as output. Experimental results show that our method has significant advantages in removing thin cloud contaminations in different bands when compared with other traditional and state-of-the-art methods.

Super-resolution (SR) of hyperspectral image (HSI) is of significance for its applications. Wavelet decomposition can be used to capture textures and structures in the HSI. In this study, we propose a multi-scale wavelet 3D convolutional neural network (MW-3D-CNN) for HSI SR. Instead of reconstructing the high resolution (HR) HSI directly, we predict the wavelet coefficients of HR HSI with the proposed network, which is composed of an embedding subnet and a predicting subnet. Both of them are built with 3D convolutional layers. The embedding subnet extracts deep spatial-spectral features from the low resolution (LR) HSI and represents the LR HSI as a set of feature cubes. The feature cubes are then fed to the predicting subnet. There are multiple output branches in the predicting subnet, each of which corresponds to a wavelet sub-band and predicts the wavelet coefficients of HR HSI. By applying inverse wavelet transform to the predicted wavelet coefficients, the HR HSI can be obtained. In the training stage, we propose to train MW-3D-CNN with L1 norm loss, which is more suitable than the conventional L2 norm loss for penalizing the errors in different wavelet sub-bands. In the experiment, the performance is tested on several HSI datasets.

A spectral super-resolution method is proposed in this paper to recover a high spectral resolution hyperspectral (HS) image from multispectral (MS) images. The proposed method involves spectral improvement strategy and spatial preservation strategy. For spectral improvement strategy, auxiliary MS/HS image pairs of different landscapes are exploited to estimate spectral response relationship so that a HS image is obtained as an intermediate result. Then, spectral dictionary learning is exploited to recover more accurate spectral reconstruction result. Spatial preservation strategy is used as spatial constraint to ensure spatial consistency. Additionally, low rank property of HS image is also introduced to make use of global spectral coherence among HS bands. Experiments are conducted on real MS/HS data (ALI and Hyperion) captured by EO-1 satellite. Experiment results demonstrate the superiority of our proposed method to other state-of-art methods.

Hyperspectral image (HSI) enjoys great advantages over more traditional image types for various applications due to the extra knowledge available. For the nonideal optical and electronic devices, HSI is always corrupted by various noises, such as Gaussian noise, deadlines, and stripings. The global correlation across spectrum (GCS) and nonlocal self-similarity (NSS) over space are two important characteristics for HSI. In this paper, a nonlocal low-rank regularized CANDECOMP/PARAFAC (CP) tensor decomposition (NLR-CPTD) is proposed to fully utilize these two intrinsic priors. To make the rank estimation more accurate, a new manner of rank determination for the NLR-CPTD model is proposed. The intrinsic GCS and NSS priors can be efficiently explored under the low-rank regularized CPTD to avoid tensor rank estimation bias for denoising performance. Then, the proposed HSI denoising model is performed on tensors formed by nonlocal similar patches within an HSI. The alternating direction method of multipliers-based optimization technique is designed to solve the minimum problem. Compared with state-of-the-art methods, the proposed algorithm can greatly promote the denoising performance of an HSI in various quality assessments.

While hyperspectral (HS) images play a significant role in many applications, they often suffer from issues such as low spatial resolution, low temporal resolution, and some of the acquired spectral bands are either with low signal-to-noise ratio (SNR) or invalid because of the very high-noise level. To address this issue, a spectral super-resolution method is proposed in this paper to recover a high-spectral-resolution HS image from multispectral (MS) images. The reconstructed HS image will have the same spatial resolution and coverage as the input MS image. The proposed method involves spectral improvement strategy and spatial preservation strategy. For spectral improvement strategy, auxiliary MS/HS image pairs of different landscapes are exploited to estimate spectral response relationship so that an HS image is obtained as an intermediate result. Then, spectral dictionary learning is exploited to recover a more accurate spectral reconstruction result. Spatial preservation strategy is used as a spatial constraint to ensure spatial consistency. In addition, the low-rank property of HS image is also introduced to make the use of global spectral coherence among HS bands. Experiments are conducted on both simulated and real datasets including spectral enhancement of RGB image and the MS image generated by AVIRIS data and real MS/HS data (ALI and Hyperion) captured by Earth Observing-1 (EO-1) satellite. Experiment results demonstrate the superiority of our proposed method to other state-of-the-art methods.

This paper presents a joint dehazing and denoising scheme for an image taken in hazy conditions. Conventional image dehazing methods may amplify the noise depending on the distance and density of the haze. To suppress the noise and improve the dehazing performance, an imaging model is modified by adding the process of amplifying the noise in hazy conditions. This model offers depth-chromaticity compensation regularization for the transmission map and chromaticity-depth compensation regularization for dehazing the image. The proposed iterative image dehazing method with polarization uses these two joint regularization schemes and the relationship between the transmission map and dehazed image. The transmission map and irradiance image are used to promote each other. To verify the effectiveness of the algorithm, polarizing images of different scenes in different days are collected. Different algorithms are applied to the original images. Experimental results demonstrate that the proposed scheme increases visibility in extreme weather conditions without amplifying the noise.

Classical matrix-based denoising methods for hyperspectral imagery (HSI) may cause spatial and spectral distortion. To improve denoising performance, a multi-linear weighted nuclear norm minimization was proposed for HSI denoising. By considering spectral continuity and inter-dependency of three unfolding modes, a multi-linear rank was proposed to model the spatial and spectral nonlocal similarity. To make the proposed method more tractable, a variable splitting based technique was used to solve the optimization problem. Experiment results reveal that the proposed method outperforms state-of-the-art methods both visually and quantitatively.

This paper proposes a novel method based on deformable dictionary learning for detecting the regions of change between multitemporal image pairs. We build on our previous work, which constructed a pair of dictionaries. The main shortcoming of this method was its dependence on a large amount of training data. In practice, there is often a shortage of ground-truthed training images, which limits the expression capability of the resulting dictionaries. This paper overcomes this challenge by incorporating the concept of deformation, wherein each atom of a dictionary is no longer a simple image patch, but instead is a flexible image deformation function. This enables the creation of more expressive dictionaries, capable of generalizing to a far greater variety of image patterns, while using a far smaller amount of ground-truthed images for supervised dictionary training. Deformation similarity is employed for patch matching to find the best set of atoms in the difference image (DI) dictionary for reconstructing image patches for a new input DI. Each such atom can be deformed to achieve a better match, thus extending generality while reducing the number of atoms needed in the dictionary. Multiple deformed atoms are weighted and combined to best reconstruct the input DI patch. Then, the same set of deformations and weights is projected to the corresponding atoms in the CD dictionary to obtain the output change-detection map. Experiments in six realistic synthetic aperture radar data sets demonstrate the robustness and efficiency of the proposed method in comparison with five other state-of-the-art methods from the literature.

Super-resolution image reconstruction has been utilized to overcome the problem of spatial resolution limitation in hyperspectral (HS) imaging. To improve the spatial resolution of HS image, this paper proposes an HS-multispectral (MS) fusion method, which exploits spatial and spectral correlations and proper regularization. High spatial correlation between MS image and the desired high-resolution HS image is conserved via an over-completed dictionary, and the spectral degradation between them projected onto the space of sparsity is applied as the spectral constraint. The high spectral correlation between high-spatial- and low-spatial-resolution HS image is preserved through linear spectral unmixing. The idea of an interactive feedback proposed in our previous work is also used when dealing with spatial reconstruction and unmixing. Low-rank property is introduced in this paper to regularize the sparse coefficients of the HS patch matrix, which is utilized as the spatial constraint. Experiments on both simulated and real data sets demonstrate that the proposed fusion algorithm achieves lower spectral distortions and the super-resolution results are superior to those of other state-of-the-art methods.

Feature extraction is of significance for hyperspectral image (HSI) classification. Compared with conventional hand-crafted feature extraction, deep learning can automatically learn features with discriminative information. However, two issues exist in applying deep learning to HSIs. One issue is how to jointly extract spectral features and spatial features, and the other one is how to train the deep model when training samples are scarce. In this paper, a deep convolutional neural network with two-branch architecture is proposed to extract the joint spectral-spatial features from HSIs. The two branches of the proposed network are devoted to features from the spectral domain as well as the spatial domain. The learned spectral features and spatial features are then concatenated and fed to fully connected layers to extract the joint spectral-spatial features for classification. When the training samples are limited, we investigate the transfer learning to improve the performance. Low and mid-layers of the network are pretrained and transferred from other data sources; only top layers are trained with limited training samples extracted from the target scene. Experiments on Airborne Visible/Infrared Imaging Spectrometer and Reflective Optics System Imaging Spectrometer data demonstrate that the learned deep joint spectral-spatial features are discriminative, and competitive classification results can be achieved when compared with state-of-the-art methods. The experiments also reveal that the transferred features boost the classification performance.

This paper presents an interactive feedback scheme of spatial resolution enhancement and spectral unmixing in hyperspectral imaging. Traditionally spatial resolution enhancement and spectral unmixing operations have been carried out separately, often in series. In such sequential processing, spatially enhanced hyperspectral images (HSIs) may introduce distortion in spectral fidelity making spectral unmixing results unreliable, or vice versa. Since both high- and low-resolution HSIs have the same endmembers, the deviation in spectral unmixing between targets and estimated high-resolution HSIs can be used as feedback to control spatial resolution enhancement. The spatial difference before and after unmixing can also be used as feedback to enhance spectral unmixing. Therefore, spectral unmixing is utilized as a constraint to spatial resolution enhancement, while spatial resolution enhancement helps improve spectral unmixing results. The performance of spatial resolution enhancement and spectral unmixing can be improved since one behaves like a prior to the other. Experimental results on both simulated and real HSI data sets demonstrate that the proposed interactive feedback scheme simultaneously achieved spatial resolution enhancement and spectral unmixing fidelity. This paper is an extended version of the previous work.

Performance of hyperspectral image classification depends on feature extraction. Compared with conventional hand-crafted feature extraction, deep learning can learn feature with more discriminative information. In this paper, a two-channel deep convolutional neural network (Two-CNN) is proposed to learn jointly spectral-spatial feature from hyperspectral image. The proposed model is composed of two channels of CNN, each of which learns feature from spectral domain and spatial domain respectively. The learned spectral feature and spatial feature are then concatenated and fed to fully connected layer to extract joint spectral-spatial feature for classification. When number of training samples is limited, we propose to train the deep model using transfer learning to improve the performance. Low-layer and mid-layer features of the deep model are learned and transferred from other scenes, only top-layer feature is learned using the limited training samples of the current scene. Experiment results on real data demonstrate the effectiveness of the proposed method.

Hyperspectral image (HSI) denoising is significant for correct interpretation. In this paper, a sparse representation framework that unifies denoising and spectral unmixing in a closed-loop manner is proposed. While conventional approaches treat denoising and unmixing separately, the proposed scheme utilizes spectral information from unmixing as feedback to correct spectral distortion. Both denoising and spectral unmixing act as constraints to the others and are solved iteratively. Noise is suppressed via sparse coding, and fractional abundance in spectral unmixing is estimated using the sparsity prior of endmembers from a spectral library. The abundance of endmembers is used as a spectral regularizer for denoising based on the hypothesis that spectral signatures obtained from a denoising process result are close to those of unmixing. Unmixing restrains spectral distortion and results in better denoising, which reciprocally leads to further improvements in unmixing. The strength of our proposed method is illustrated by simulated and real HSIs with performance competitive to the state-of-the-art denoising and unmixing methods.

Airborne laser scanning (ALS) data are an important source of information for forest inventory purposes. In particular they allow us to delineate individual tree crowns (ITC) that are at the basis of the individual tree-based inventories. In multi-layered forests various tree species are mixed together and trees usually grow in a different vertical layers, leading to a relevant problem in detecting subdominant and suppressed trees. Thus, the purpose of this study is to present an approach for ITC delineation using clustering techniques at both 2D and 3D level based on raw ALS point cloud. The preliminary results showed that forest structure strongly affect the performance of the proposed algorithm. Thus, different criteria were chosen with a priori knowledge from ground truth data. The proposed algorithm achieved comparable or superior results as compared to conventional methods.

The acquired hyperspectral data are always in low resolution in both spatial and spectral domains, which will result in lots of mixed pixels and degrade the detection and recognition performance in civil and military applications. So many super resolution techniques are applied to overcome this limit. In this paper, we propose a coupled hyperspectral spatial super-resolution and spectral unmixing method based on sparse representation. Combing spatial super-resolution and spectral unmixing can precisely conserve both spatial information and spectral correlation among different bands. Spectral unmixing is taken as a regularization term in spatial super-resolution to test spectral consistency and avoid spectral distortion, while spatial super-resolution is used to enhance the resolution of abundance map after spectral unmixing.

This paper presents a validation of a recently developed method for simulating new spectral bands from multispectral images. We use the method to simulate CHRIS/PROBA bands from a Landsat ETM+ image. Next we validate the simulated bands by RMSE, band-band scatter plots and visual comparison with original CHRIS bands acquired 4 days before the ETM+ image. Finally, we calculate NDVI from the original and simulated CHRIS bands and from the ETM+ bands. Result show that the simulated CHRIS NDVI in the dominant vegetation types in the study area is more correlated to the original CHRIS NDVI, than ETM+ NDVI. For not dominant vegetation types, not included into the simulation process, the correlation is lower for simulated CHRIS NDVI than for ETM+ NDVI.

This work extends our previous work on risk estimation of a mine contaminated area using mine action information such as minefield records and mine incidence, and feature indicators extracted from satellite images. Changes in temporal NDVI's and access to main transport routes and protective landscape such as forest edge are used to indicate potentially risk areas. All features are harmonized in an GIS environment. Kernel density function is performed on landmine and Earth Observation indicators. Safe areas are identified by land-in-use. A final risk map is produced by merging safe areas and risk estimation layer.

Multiple Endmember Spectral Mixture Analysis (MESMA) is a useful method to deal with issues of spatial and spectral heterogeneity in urban areas, allowing endmembers to vary on a per-pixel basis. In this study, per-pixel land-cover fractions obtained from MESMA are used as an input for spatially distributed hydrological modeling in a strongly urbanized area. Different components of the water balance are estimated using per-pixel land-cover fractions and compared to results obtained with a semi-distributed approach, where land-cover composition is fixed a priori for each land-use type, using default model parameters. Access to spatially detailed information on the impervious surface and vegetation component proves to have a strong impact on surface runoff and groundwater recharge calculations for different land-uses. Use of MESMA, providing accurate estimates land-cover composition for each pixel may therefore contribute to improving water balance calculation in urbanized areas.

We present a novel approach for spectral unmixing in hyperspectral imagery called local linear spectral unmixing (LLSU). Our new proposal relies on an existing strategy for non-linear data modelling where a general non-linear model is approximated via several piecewise linear models. The algorithm is the result of hybridizing two well-known strategies for exploratory high-dimensional data analysis: cluster analysis and non-negative matrix factorization. It has been proposed to answer the limitations of global linear unmixing models which are widely used for spectral unmixing. Our strategy is to first group similar pixels from the hyperspctral image via cluster analysis and then to consider a local linear unmixing model for each cluster in order to obtain the constituent endmembers. Subsequently, the resulted local endmembers from each cluster of mixed pixels are at their turn clustered based on their spectral similarity; the final solution is given by the clusters' centroids.

An extended superresolution observation model is proposed for POCS superresolution of hyperspectral images. Multiple constraint criteria based on a priori knowledge were incorporated: data consistence, amplitude constraint, Total Variation edge smoothing constraint, outlier rejection, and PCA based denoising. The constraint criteria are applied using POCS superresolution reconstruction. The method was tested with both simulation and multi-viewing hyperspectral CHRIS images. Preliminary results of the constraint based superresolution shows potential for angular hyperspectral images.

In this paper, the potential of Compact High-Resolution Imaging Spectrometer (CHRIS)/Project for On-Board Autonomy data for impervious surface mapping is tested in a mixed urban/suburban/rural environment including part of the city of Leuven (Belgium) using multiple endmember unmixing. Various unmixing scenarios are compared, using different threshold values for theRMSEcriterion applied to select the proper model for unmixing each pixel. Validation based on 25-cm aerial photography shows that the use of threshold values that favor the application of models with a small number of endmembers performs better compared to scenarios that make use of models with more endmembers. Detailed analysis of model selection for pixels with different land-cover composition indicates that the error in fraction estimation is partly related to spectral confusion between impervious surface types and bare soil, leading to the selection of inappropriate models for the unmixing. In spite of the spectral similarity of soil and impervious surface endmembers, average fractional error for impervious surfaces, vegetation, and bare soil is around 15%, which demonstrates the potential of CHRIS data for mapping the major physical components of the urban/suburban environment at the subpixel scale.

Natura 2000 is an ecological network of protected areas in the territory of the European Union (EU). With the introduction of the Habitats Directive in 1992, EU member states are obligated to report every six years the status of the Natura 2000 habitats so that better conservation policy can be formulated. This paper examines the use of angular hyperspectral CHRIS/Proba image for the mapping of heathland at a Belgian Natura 2000 site. We find that the use of angular images increases the overall classification rate as compared to using only the nadir image; with the incorporation of angular images the final mapping is also more homogenous with less salt and pepper effect. While the class accuracy of Calluna- and Erica-dominated heathlands are still low, class accuracy of Molinia-dominated heathland is generally more encouraging. Two tree-based ensemble classifiers, Random Forest (RF) and Adaboost, were compared with Support Vector Machines (SVM). When only the nadir image was used, SVM attained the highest accuracy. When angular images were included, all three classifiers obtained comparable accuracies though in general RF and Adaboost had faster training time. We also adopted an assessment approach which repeats the accuracy assessment in ten independent trials, instead of the common practice of having only one trial. Our results show that accuracy attainment can vary significantly among different trials and hence it is recommendable to have more than one trial in order that a more objective characterization of the classifiers is obtained.1

Superresolution (SR) image reconstruction is a technique to obtain a high-resolution (HR) image from a set of low-resolution images. Compact High Resolution Imaging Spectrometer (CHRIS)/Project for On-Board Autonomy (Proba) is a transitional hyperspectral-oriented satellite which acquires multiple angular images of the same scene. The angular images acquired within a very short period of time are ideal for SR operation. Recent developments point to the possibility of a SR-enhanced CHRIS data set at higher resolution. Apparently, the additional details are valuable information for many applications. This letter presents the preliminary evaluation of the SR CHRIS images for ecotope mapping and subpixel classification of sealed surface using two different scenes in Belgium. Accuracy obtained from SR CHRIS images is comparable to that of the original CHRIS, but with significantly more detail in the final classification map. In view of the demands for HR hyperspectral data sets, SR operation can be an interesting option to mitigate the lower spatial resolution of the current and future spaceborne hyperspectral images. Properties such as quick revisiting time and angular acquisition of a hyperspectral satellite are important for the success of SR operations.

Given the hyperspectral-oriented waveband configuration of multiangular CHRIS/Proba imagery, the scope of its application could widen if the present 18-m resolution would be improved. The multiangular images of CHRIS could be used as input for superresolution (SR) image reconstruction. A critical procedure in SR is an accurate registration of the low-resolution images. Conventional methods based on affine transformation may not be effective given the local geometric distortion in high off-nadir angular images. This paper examines the use of a nonrigid transform to improve the result of a nonuniform interpolation and deconvolution SR method. A scale-invariant feature transform is used to collect control points (CPs). To ensure the quality of CPs, a rigorous screening procedure is designed: 1) an ambiguity test; 2) them-estimator sample consensus method; and 3) an iterative method using statistical characteristics of the distribution of random errors. A thin-plate spline (TPS) nonrigid transform is then used for the registration. The proposed registration method is examined with a Delaunay triangulation-based nonuniform interpolation and reconstruction SR method. Our results show that the TPS nonrigid transform allows accurate registration of angular images. SR results obtained from simulated LR images are evaluated using three quantitative measures, namely, relative mean-square error, structural similarity, and edge stability. Compared to the SR methods that use an affine transform, our proposed method performs better with all three evaluation measures. With a higher level of spatial detail, SR-enhanced CHRIS images might be more effective than the original data in various applications.

Subpixel image registration is the key to successful image fusion and superresolution enhancement of multiangle satellite data. Multiangle image registration poses two main challenges: 1) Images captured at large view angles are susceptible to resolution change and blurring, and 2) local geometric distortion caused by topographic effects and/or platform instability may be important. In this paper, we propose a two-step nonrigid automatic registration scheme for multiangle satellite images. In the first step, control points (CPs) are selected in a preregistration process based on the scale-invariant feature transform (SIFT). However, the number of CPs obtained in this first step may be too few and/or CPs may be unevenly distributed. To remediate these problems, in a second step, the preliminary registered image is subdivided into chips of 64 × 64 pixels, and each chip is matched with a corresponding chip in the reference image using normalized cross correlation (NCC). By doing so, more CPs with better spatial distribution are obtained. Two criteria are applied during the generation of CPs to identify outliers. Selected SIFT and NCC CPs are used for defining a nonrigid thin-plate-spline model. The proposed registration scheme has been tested using data from the Compact High Resolution Imaging Spectrometer (CHRIS) onboard the Project for On-Board Autonomy (Proba) satellite. Experimental results demonstrate that the proposed method works well in areas with little variation in topography. Application in areas with more pronounced relief would require the use of orthorectified image data in order to achieve subpixel registration accuracy.

Iterative backprojection superresolution (SR) image enhancement was applied to single-date and multi-date hyperspectral CHRIS imagery. SR enhanced imagery provides substantial increase in image detail and better characterization of image objects, in particular man-made objects such as buildings and roads. In terms of increase in signal to noise ratio, bi-temporal input produces only marginally better results than single-date input. However, better contrast in transition zones of rural habitat types is observed with the former. SR enhanced CHRIS imagery could potentially broaden the scope of application of this type of hyperspectral-oriented image data.

The snowline altitude (SLA) and the accumulation area ratio (AAR) of the Morteratsch glacier, Switzerland are derived using Landsat images over a period of 20 year. To draw the SLA, multitemporal Landsat images are first calibrated to surface reflectance using 6S [1]. A linear spectral unmixing algorithm is applied with accumulation and ablation end-members. Transects best representing the morphology of the glacier are drawn and the SLA is defined using the shifts between the end-member profiles of snow and ice. The results of two mass balance characteristics, SLA and AAR, show that the Morteratsch glacier has changed substantially during the period between 1985 and 2005. The average SLA of the glacier has risen by 131 m and the AAR decreased from 66.2 % to 52.5 % during this period. Comparatively, the eastern part of the Morteratsch glacier has a smaller increase (94 m) in the altitude of the snowline, as compared to that of the western part (183 m).

We investigated two binary classification strategies to further extend the strength of ensemble classifiers for mapping of urban objects. The first strategy was a one-against-one approach. The idea behind it was to employ a pairwise binary classification where n(n-1)/2 classifiers are created, n being the number of classes. Each of the n(n-1)/2 classifiers was trained using only training cases from two classes at a time. The ensemble was then combined by majority voting. The second strategy was a one-against-all binary approach: if there are n classes, with a = {1,..., n} being one of the classes, then n classifiers were generated, each representing a binary classification of a and non-a. The ensemble was combined using accuracy estimates obtained for each class. Both binary strategies were applied on two single classifiers (decision trees and artificial neural network) and two ensemble classifiers (Random Forest and Adaboost). Two multi-source data sets were used: one was prepared for an object-based classification and one for a conventional pixel-based approach. Our results indicate that ensemble classifiers generate significantly higher accuracies than a single classifier. Compared to a single C5.0 tree, Random Forest and Adaboost increased the accuracy by 2 to 12%. The range of increase depends on the data set that was used. Applying binary classification strategies often increases accuracy, but only marginally (between 1-3%). All increases are statistically significant, except on one occasion. Coupling ensemble classifiers with binary classification always yielded the highest accuracies.

This paper discusses the application of superresolution (SR) image reconstruction on multi-angle Chris/Proba images. The goal is to increase the spatial resolution of Chris/Proba images, with 18 bands from 0.4-1.0 mum in the hope to obtain a better ecotope classification. The SR approach chosen for this study is Total Variation, an iterative method which models the relationship between the desired high resolution image and the low resolution images, with the following components: a subsampling factor, a point spread function, an estimated rotation and shift, and a regularization term. This regularization approach is fast in implementation and flexible in handling noise. Efficient gradient descent methods can be used to find the desired high resolution image. The spatial resolution of the original image is improved from 25 m to 12 m using Total Variation. Subjective assessment through visual interpretation shows substantial improvement in detail. A tree-based ensemble classifier Random Forest is used for the classification of 18 ecotopes. Overall accuracy shows a 10% increase with the SR derived Chris/Proba images, compared with a classification based on the original imagery. Our results demonstrate that SR methods can improve spatial detail of multi-angle images, and subsequently classification accuracy.

Meter to submeter resolution satellite images have generated new interests in extracting man-made structures in the urban area. However, classification accuracies for such purposes are far from satisfactory. Spectral characteristics of urban land cover classes are so similar that they cannot be separated using only spectral information. As a result, there is an increased interest in incorporating geometrical information. One possible approach is the use of morphological profiles (MPs). In this paper, we introduce two improvements on the use of MPs. Current approaches use disk-shaped structuring elements (SEs) to derive an MP. This profile contains information about the minimum dimension of objects. In this paper, we extend this approach by using linear SEs. This results in a profile containing information about the maximum object dimension. We show that the addition of the line-based MP gives a substantial improvement of the classification result. A second improvement is achieved by using ldquopartial morphological reconstructionrdquo instead of the normal morphological reconstruction. Morphological reconstruction is commonly used to better preserve the shape of objects. However, we show that this leads to ldquoover-reconstructionrdquo in typical remote sensing images and a decreased classification performance. With ldquopartial reconstruction,rdquo we are able to overcome this problem and still preserve the shape of objects.

Meter to sub-meter resolution satellite images have generated new interests in extracting man-made structures in the urban area. However, classification accuracies for such purposes are far from satisfactory. Spectral characteristics of urban land cover classes are so similar that they cannot be separated using only spectral information. As a result, there is an increased interest in incorporating geometrical information. One possible approach is the use of a morphological profile [1]. This profile contains information about the size of objects. In literature this is usually combined with morphological reconstruction to better preserve the shapes of objects. In this paper, we show that when used for remote sensing images this leads to ‘overreconstruction’, with a decreased classification performance as a result. We propose a new method called ‘partial reconstruction’ to overcome this problem and still be able to preserve the shape of objects. Classification experiments show a better performance with partial reconstruction.

This paper investigated a probabilistic method using likelihood and granularity measures for scale selection in the case of anisotropic diffusion filtering. A multi-scale stack of diffused images is created, from which the optimal scale for subsequent analysis is extracted. For each pixel at each scale, a likelihood measure based upon the minimal description length principles is attached. By maximizing the likelihood that each pixel is optimal represented at a certain scale, a local scale map can be defined. The latter is subsequently augmented by a MRF- related granularity measure (Ising potential). This ensures the retainment of certain localized details. The global scale is defined as the scale at which the difference between the corresponding diffused image and the local scale image is minimal. The proposed approach is applied as a preprocessing step for the classification of high resolution air-borne multi-spectral images. Initial results show that the probabilistic scale selection identifies a suitable ideal scale.

Two ensemble methods, bagging and boosting, were investigated for improving algorithm performance. The authors' results confirmed the theoretical explanation of L. Breiman (1996) that bagging improves unstable, but not stable, learning algorithms. While boosting enhanced accuracy of a weak learner, its behavior is subject to the characteristics of each learning algorithm.