Deep learning-based unmixing methods have received great attention in recent years and achieved remarkable performance. These methods employ a data-driven approach to extract structure features from hyperspectral images; however, they tend to be less physically interpretable. Conventional unmixing methods have much more interpretability, whereas they require manually designing regularization and choosing penalty parameters. To overcome these limitations, we propose a novel unmixing method by unrolling the plug-and-play unmixing algorithm to conduct the deep architecture. Our method integrates both inner and outer priors. The carefully designed unfolding deep architecture is used to learn the spectral and spatial information from the hyperspectral image, which we refer to as inner priors. Additionally, our approach incorporates deep denoisers that have been pretrained on a large volume of image data to leverage the outer priors. Second, we design a dynamic convolution to model the multiscale information. Different scales are fused with an attention module. Experimental results of both synthetic and real datasets demonstrate that our method outperforms compared methods.

Hyperspectral unmixing aims to decompose the mixed pixels into pure spectra and calculate their corresponding fractional abundances. It holds a critical position in hyperspectral image processing. Traditional model-based unmixing methods use convex optimization to iteratively solve the unmixing problem with hand-crafted regularizers. While their performance is limited by these manually designed constraints, which may not fully capture the structural information of the data. Recently, deep learning-based unmixing methods have shown remarkable capability for this task. However, they have limited generalizability and lack interpretability. In this paper, we propose a novel hyperspectral unmixing method regularized by a diffusion model (URDM) to overcome these shortcomings. Our method leverages the advantages of both conventional optimization algorithms and deep generative models. Specifically, we formulate the unmixing objective function from a variational perspective and integrate it into a diffusion sampling process to introduce generative priors from a denoising diffusion probabilistic model (DDPM). Since the original objective function is challenging to optimize, we introduce a splitting-based strategy to decouple it into simpler subproblems. Extensive experiment results conducted on both synthetic and real datasets demonstrate the efficiency and superior performance of our proposed method.

Spectral unmixing has been extensively studied with a variety of methods and used in many applications. Recently, data-driven techniques with deep learning methods have obtained great attention to spectral unmixing for its superior learning ability to automatically learn the structure information. In particular, autoencoder based architectures are elaborately designed to solve blind unmixing and model complex nonlinear mixtures. Nevertheless, these methods perform unmixing task as black-boxes and lack interpretability. On the other hand, conventional unmixing methods carefully design the regularizer to add explicit information, in which algorithms such as plug-and-play (PnP) strategies utilize off-the-shelf denoisers to plug powerful priors. In this paper, we propose a generic unmixing framework to integrate the autoencoder network with regularization by denoising (RED), named AE-RED. More specially, we decompose the unmixing optimized problem into two subproblems. The first one is solved using deep autoencoders to implicitly regularize the estimates and model the mixture mechanism. The second one leverages the denoiser to bring in the explicit information. In this way, both the characteristics of the deep autoencoder based unmixing methods and priors provided by denoisers are merged into our well-designed framework to enhance the unmixing performance. Experiment results on both synthetic and real data sets show the superiority of our proposed framework compared with state-of-the-art unmixing approaches.

When adopting a model-based formulation, solving inverse problems encountered in multiband imaging requires to define spatial and spectral regularizations. In most of the works of the literature, spectral information is extracted from the observations directly to derive data-driven spectral priors. Conversely, the choice of the spatial regularization often boils down to the use of conventional penalizations (e.g., total variation) promoting expected features of the reconstructed image (e.g., piece-wise constant). In this work, we propose a generic framework able to capitalize on an auxiliary acquisition of high spatial resolution to derive tailored data-driven spatial regularizations. This approach leverages on the ability of deep learning to extract high level features. More precisely, the regularization is conceived as a deep generative network able to encode spatial semantic features contained in this auxiliary image of high spatial resolution. To illustrate the versatility of this approach, it is instantiated to conduct two particular tasks, namely multiband image fusion and multiband image inpainting. Experimental results obtained on these two tasks demonstrate the benefit of this class of informed regularizations when compared to more conventional ones.

The inherent spectral variability in hyperspectral images, the noise, and other factors bring difficulties to traditional detectors to separate the target and background by using linear decision boundaries. In this paper, by generalizing the classical constrained energy minimization (CEM) method, and considering the feature auto-extraction ability of deep neural networks (DNN), we propose a nonlinear detector based on semi-supervised learning (named deepCEM). This approach designs a deep neural network structure to provide a specific form of the nonlinear detector and trains the DNN model with knowledge of target spectra and unlabeled samples. Experiments performed on several hyperspectral data sets show that the proposed method performs better than other state-of-the-art methods.

Properly exploiting image properties is crucial for boosting the hyperspectral unmixing performance. Recent advanced image processing methods use deep architectures to learn image priors. However, these deep priors take effect in an implicit manner and it is nontrivial to characterize their properties. Introducing extra regularization terms is an explicit way of encoding image priors, and the plug-and-play technique enables to construct priors from data by denoisers. In this work, we propose a new unmixing framework to combine both the deep image priors (DIP) and plug-and-play (PnP) priors to further enhance the unmixing performance. The alter-nating direction method of multipliers (ADMM) framework is used to separate the optimization problem into two subproblems. The first one is solved using a U-net training step to obtain DIP, and a proximal denoising step is then used to solve the second subproblem to add denoiser priors. Experiment results demonstrate the effectiveness of our proposed method.

Jointly exploiting spectral information and spatial information, rather than working on individual pixels, is important for hyperspectral target detection. In this letter, we propose a hyperspectral target detection method relying on superpixel structures of the input image. Multiscale superpixels are generated to capture textures of the image, and each superpixel is summarized to its representative, which is the average of all its pixels. The SparseCEM detector is then applied to these representatives. Finally, the detection results from all scales are fused to achieve the final output. Our experiment results show that the multiscale-superpixel-based SparseCEM detector (MSSD) outperforms the compared typical detection methods.

Nowadays, nonnegative matrix factorization (NMF)-based methods have been widely applied to blind spectral unmixing. Introducing proper regularizers to NMF is crucial for mathematically constraining the solutions and physically exploiting spectral and spatial properties of images. Generally, properly handcrafted regularizers and solving the associated complex optimization problem are nontrivial tasks. In our work, we propose an NMF-based unmixing framework which jointly uses a learned regularizer from data and a handcrafted regularizer. To be specific, we plug learned priors of abundances where the associated subproblem can be addressed using various image denoisers, and we consider an$\ell _{2,1}$-norm as an example to illustrate the way of integrating handcrafted regularizers. The proposed framework is flexible and extendable. Both synthetic data and real airborne data are conducted to confirm the effectiveness of our method.

Recently, the use of a deep autoencoder-based method in blind spectral unmixing has attracted great attention as the method can achieve superior performance. However, most autoencoder-based unmixing methods use non-structured reconstruction loss to train networks, leading to the ignorance of band-to-band-dependent characteristics and fine-grained information. To cope with this issue, we propose a general perceptual loss-constrained adversarial autoencoder network for hyperspectral unmixing. Specifically, the adversarial training process is used to update our framework. The discriminate network is found to be efficient in discovering the discrepancy between the reconstructed pixels and their corresponding ground truth. Moreover, the general perceptual loss is combined with the adversarial loss to further improve the consistency of high-level representations. Ablation studies verify the effectiveness of the proposed components of our framework, and experiments with both synthetic and real data illustrate the superiority of our framework when compared with other competing methods.

The complex nature of hyperspectral images makes the analysis of spectral signatures a challenging task in remote sensing. For quantitative analysis, spectral unmixing is a well-established and effective tool to analyze the spectra and spatial distribution of substances in the scene. The classical unmixing algorithms usually fail to tackle spectral variability caused by variations in environmental conditions. Many variants based on the linear mixing process have been proposed to tackle this problem; however, the spectral variability modeling capacity of these algorithms is usually insufficient. In this article, we present a probabilistic generative model to address endmember variability and provide more accurate abundance and endmember estimates. The proposed model simultaneously extracts the endmembers and estimates abundances in an unsupervised manner. In particular, it allows fitting arbitrary endmember distributions through the nonlinear modeling capability of neural networks compared to other methods that use parametric endmember variability models. The performance of the proposed approach is evaluated on both synthetic and real datasets. Experimental results show its superiority in comparison with other state-of-the-art methods. The code of this work is available athttps://github.com/shuaikaishi/PGMSUfor the sake of reproducibility.

Hyperspectral unmixing plays an important role in hyperspectral image processing and analysis. It aims to decompose mixed pixels into pure spectral signatures and their associated abundances. The hyperspectral image contains spatial information in neighborhood regions, and spectral signatures existing in the region also have a high correlation. However, most autoencoder (AE)-based unmixing methods are pixel-to-pixel methods and ignore these priors. It is helpful to add spectral–spatial information into unmixing methods. A recent trend to deal with this problem is to use convolutional neural networks (CNNs). Our proposed framework uses 3-D-CNN-based networks to jointly learn spectral–spatial priors. Moreover, previous AE-based unmixing methods use fixed spectral signatures for each pure material. In our work, we use a carefully designed decoder to cope with the endmember variability issue, and variational inference strategy is applied to add uncertainty property into endmembers. To avoid overfitting, we use structured sparsity regularizers to the encoder networks, and$\ell _{2,1}$-loss is added to the estimated abundances to guarantee the sparseness. Experimental results on both simulated and real data demonstrate the effectiveness of our proposed method.

Spectral unmixing is a widely used technique in hyperspectral image processing and analysis. It aims to separate mixed pixels into the component materials and their corresponding abundances. Early solutions to spectral unmixing are performed independently on each pixel. Nowadays, investigating proper priors into the unmixing problem has been popular as it can significantly enhance the unmixing performance. However, it is nontrivial to handcraft a powerful regularizer, and complex regularizers may introduce extra difficulties in solving optimization problems in which they are involved. To address this issue, we present a plug-and-play (PnP) priors framework for hyperspectral unmixing. More specifically, we use the alternating direction method of multipliers (ADMM) to decompose the optimization problem into two iterative subproblems. One is a regular optimization problem depending on the forward model, and the other is a proximity operator related to the prior model and can be regarded as an image denoising problem. Our framework is flexible and extendable which allows a wide range of denoisers to replace prior models and avoids handcrafting regularizers. Experiments conducted on both synthetic data and real airborne data illustrate the superiority of the proposed strategy compared with other state-of-the-art hyperspectral unmixing methods.

Spectral unmixing is an important task in hyperspectral image processing for separating the mixed spectral data pertaining to various materials observed aiming at analyzing the material components in observed pixels. Recently, nonlinear spectral unmixing has received particular attention in hyperspectral image processing, as there are many situations in which the linear mixture model may not be appropriate and could be advantageously replaced by a nonlinear one. Existing nonlinear unmixing approaches are often based on specific assumptions on the nonlinearity and can be less effective when used for scenes with unknown nonlinearity. This article presents an unsupervised nonlinear spectral unmixing method that addresses a general model that consists of a linear mixture part and an additive nonlinear mixture part. The structure of a deep autoencoder network, which has a clear physical interpretation, is specifically designed to achieve this purpose. Moreover, a convolutional neural network (CNN) is used to capture the spectral-spatial priors from hyperspectral data. Extensive experiments with synthetic and real data illustrate the generality and effectiveness of this scheme compared with state-of-the-art methods.

Spectral signatures are usually affected by variations in environmental conditions. The spectral variability is thus one of the most important and challenging problems to be addressed in hyperspectral unmixing. Generally, it is a non-trivial task to model the endmember variability, and existing spectral unmixing methods that address the spectral variability have different limitations. This paper presents a variational autoencoder (VAE) framework for hyperspectral unmixing accounting for the endmember variability. The endmembers are generated using the posterior distributions of the latent variables to describe their variability in the image. Compared with other existing distribution based methods, the proposed method is able to fit an arbitrary distribution of endmembers for each material through the representation capacity of deep neural networks. Evaluated with both synthetic and real datasets, the proposed method shows superior unmixing results compared with other state-of-the-art unmixing methods.

Removing shadows that are often present in remotely sensed hyperspectral images is important for both enhancing the interpretability of the data and further target analysis. Shadow removal approaches based on spectral unmixing have been proposed in the literature using the linear mixture model. However, objects that produce shadows may also introduce light scattering, and the higher order interactions of photons can cause nonlinearity. This letter integrates the nonlinear hyperperspectral unmixing into the unmixing-based shadow removal, and the effects of applying typical nonlinear algorithms within the approach are investigated. The usefulness of nonlinear unmixing in hyperspectral shadow removal is verified based on the results of applications to both laboratory-created real data and actual airborne data.

High spectral resolution of hyperspectral images allows the detection and classification of materials in the observed images. However, existing research on hyperspectral detection mainly focuses on pixel-level study, partially due to the low spatial resolution in typical earth observation applications. With the development of imaging techniques, high-spatial-resolution hyperspectral data can be obtained and object-level detection is necessary for many applications. In this work, the object-based hyperspectral detection problem is formulated, and a convolutional neural network is then designed based on the specific characteristics of this problem. Moreover, a hyperspectral dataset with over 400 high-quality images for object-level target detection is created. Experimental results validate the proposed framework and show its superior performance.

Hyperspectral unmixing aims at separating a mixed pixel into a set of pure spectral signatures and their corresponding fractional abundances. Investigating prior spatial and spectral information to regularize the unmixing problem can effectively improve the estimation performance. However, handcrafting a powerful regularizer is a non-trivial task and complex regularizers introduce extra difficulties in solving the optimization problem. In this paper, we present a flexible spectral unmixing method using plug-and-play priors. This method benefits from the alternating direction method of multipliers (ADMM) to decompose the optimization problem into iterative subproblems and incorporates the image denoisers as prior models in a subproblem. In this form, we can plug in various image denoising operations to bypass handcrafting regularizers. We demonstrate the superiority of the proposed unmixing method comparing with other state-of-the-art methods both on synthetic data and real airborne data.

Near-infrared (NIR) images provide spectral information beyond the visible light spectrum and thus are useful in many applications. However, single-channel NIR images contain less information per pixel than RGB images and lack visibility for human perception. Transforming NIR images to RGB images is necessary for performing further analysis and computer vision tasks. In this work, we propose a novel NIR-to-RGB translation method. It contains two sub-networks and a fusion operator. Specifically, a U-net based neural network is used to learn the texture information while a CycleGAN based neural network is adopted to excavate the color information. Finally, a guided filter based fusion strategy is applied to fuse the outputs of these two neural networks. Experiment results show that our proposed method achieves superior NIR-to-RGB translation performance.

Recently, face recognition systems have received significant attention, and there have been many works focused on presentation attacks (PAs). However, the generalization capacity of PAs is still challenging in real scenarios, as the attack samples in the training database may not cover all possible PAs. In this paper, we propose to perform the face presentation attack detection (PAD) with multi-channel images using the convolutional neural network based anomaly detection. Multi-channel images endow us with rich information to distinguish between different mode of attacks, and the anomaly detection based technique ensures the generalization performance. We evaluate the performance of our methods using the wide multi-channel presentation attack (WMCA) dataset.

Nonlinear spectral unmixing is an important and challenging problem in hyperspectral image processing. Classical nonlinear algorithms are usually derived based on specific assumptions on the nonlinearity. In recent years, deep learning shows its advantage in addressing general nonlinear problems. However, existing ways of using deep neural networks for unmixing are limited and restrictive. In this letter, we develop a novel blind hyperspectral unmixing scheme based on a deep autoencoder network. Both encoder and decoder of the network are carefully designed so that we can conveniently extract estimated endmembers and abundances simultaneously from the nonlinearly mixed data. Because an autoencoder is essentially an unsupervised algorithm, this scheme only relies on the current data and, therefore, does not require additional training. Experimental results validate the proposed scheme and show its superior performance over several existing algorithms.

Spectral unmixing is one of the most important issues of hyperspectral data processing. However, the lack of publicly available dataset with ground-truth makes it difficult to evaluate and compare the performance of unmixing algorithms. In this work, we create several experimental scenes in our laboratory with controlled settings where the pure material spectra and material compositions are known. Lab-made hyperspectral datasets with these scenes are then provided, and mutually validated with typical linear and nonlinear unmixing algorithms.