Hyperspectral imaging (HSI) captures abundant spectral information of land covers while light detection and ranging (LiDAR) provides elevation and structural characteristics. Joint classification of HSI and LiDAR data can effectively merge spectral and elevation information to enhance the outcome of land cover classification. Current HSI and LiDAR joint classification approaches mainly employ a three-layer deep network to extract high-order features, followed by a concatenation or weighted fusion scheme which cannot fully exploit the unique properties of different data modalities. Meanwhile, these methods usually require high computational resources. To alleviate these issues, this paper proposes a masked self-attention fusion network (MSAF) for joint HSI and LiDAR classification, where a cascaded cross-attention fusion framework is designed to fully merge different stages of features. First, a mobile convolution block is developed to extract multi-modal data features. Then, a multi-view sequence embedding method is proposed to effectively integrate elevation information and spectral-spatial information so as to obtain token sequences. Finally, an effective masked self-attention mechanism is designed to fuse token sequences. Experimental results on multiple datasets indicate that the proposed framework significantly outperforms other advanced multi-modal fusion methods in terms of classification performance and computing efficiency. The code of this manuscript is available on https://github.com/lulushh/MSAF.

The development of continual learning (CL) methods, which aim to learn new tasks in a sequential manner from the training data acquired continuously, has gained great attention in remote sensing (RS). The existing CL methods in RS, while learning new tasks, enhance robustness towards catastrophic forgetting. This is achieved by using a large number of labeled training samples, which is costly and not always feasible to gather in RS. To address this problem, we propose a novel continual self-supervised learning method in the context of masked autoencoders (denoted as CoSMAE). The proposed CoSMAE consists of two components: i) data mixup; and ii) model mixup knowledge distillation. Data mixup is associated with retaining information on previous data distributions by interpolating images from the current task with those from the previous tasks. Model mixup knowledge distillation is associated with distilling knowledge from past models and the current model simultaneously by interpolating their model weights to form a teacher for the knowledge distillation. The two components complement each other to regularize the MAE at the data and model levels to facilitate better generalization across tasks and reduce the risk of catastrophic forgetting. Experimental results show that CoSMAE achieves significant improvements of up to 4.94% over state-of-the-art CL methods applied to MAE. Upon acceptance of the letter, the code will be made publicly available at: https://git.tu-berlin.de/rsim/CoSMAE.

Hyperspectral unmixing, an essential and fundamental task in remote sensing, focuses on estimating endmembers (spectrally pure components) and their fractional abundances within each mixed pixel of a hyperspectral image. With the advent of deep learning (DL), the field of hyperspectral unmixing has made significant progress. Among DL approaches, autoencoder-based models have shown promising results. However, most unmixing methods estimate the endmembers by the weights of the linear layers in the decoder of their networks, making their performance highly dependent on weight initialization. Moreover, noise is not explicitly accounted for in most recent methods that use spectral angle distance (SAD) loss. To avoid the initialization problems, we developed an innovative inversion strategy to directly estimate the endmembers. Moreover, to optimally account for noise, an end-to-end network is proposed that integrates both denoising and unmixing. Finally, for an improved feature extraction, a novel spectral-spatial attention module is integrated in the network. Extensive experiments on a synthetic and three real datasets show that the proposed method significantly and consistently outperforms the compared state-of-the-art methods. The full code is available at https://github.com/xuanwentao for public evaluation.

Accurate estimation of the fractional abundances of intimately mixed materials from spectral reflectances is generally hard due to a highly nonlinear relationship between the measured spectrum and the composition of the material. Changes in the acquisition and the illumination conditions cause variability in the spectral reflectance, further complicating the spectral unmixing procedure. In this work, we propose a methodology for unmixing intimate mixtures that can tackle both nonlinearity and spectral variability. A supervised approach is proposed that characterizes the nonlinear data manifolds by high-dimensional Bézier surfaces. To deal with spectral variability, a manifold transformation procedure is designed. To generate Bézier surfaces, training samples are required that are uniformly distributed throughout the data manifold. For this, we recently generated a hyperspectral dataset of intimate mineral powder mixtures by homogeneously mixing five different clay powders (Kaolin, Roof clay, Red clay, mixed clay, and Calcium hydroxide) in laboratory settings. In total 330 samples (325 mixtures and five pure materials) were prepared. The ground fractional abundances of these mixtures uniformly cover the 5-D probability simplex. The spectral reflectances of these samples were acquired by multiple sensors with a large variation in sensor types, platforms, and acquisition conditions. Experiments are conducted both on simulated and real intimate mineral powder mixtures. Comparison with a number of unsupervised unmixing methods demonstrates the potential of the proposed approach.

Hyperspectral image (HSI) classification is commonly influenced by convolution neural networks (CNNs). However, the large number of parameters and computational complexity associated with CNNs can limit their practical application, particularly when computing and storage resources are limited. To address this challenge, we propose a channel-layer-oriented lightweight network for HSI classification. Motivated by existing structures that typically set large channels and stack multiple layers, we give more optimal solutions strategically to further compress the model. For intralayer feature extraction, we develop a channel-oriented spectral–spatial module (COS2M), which introduces a dual-single-channel (DSC) 3-D convolution that works in conjunction with depthwise convolution to fully extract spectral–spatial information. For interlayer information transmission, we propose a novel neighbor-pixel-aware activation function (NPAF), where the activation of a single pixel is determined by the learnable interaction with its neighbor range that enhances information transmission and improves the network’s fitting ability through the single activation layer. By implementing these strategies, we aim to overcome the limitations of traditional CNNs and enable efficient HSI classification within resource-constrained environments. The whole network is designed to be a compact end-to-end structure. It achieves better classification performance than other deep learning methods and lightweight models, even with limited training samples. The network parameters, model complexity, and inference time also demonstrate significant superiority, as confirmed by experiments on three benchmark datasets. The source codes are available publicly at:https://github.com/AchunLee/CLOLN_TGRS

Spectral pixels are often a mixture of the pure spectra of the materials, called endmembers, due to the low spatial resolution of hyperspectral sensors, double scattering, and intimate mixtures of materials in the scenes. Unmixing estimates the fractional abundances of the endmembers within the pixel. Depending on the prior knowledge of endmembers, linear unmixing can be divided into three main groups: supervised, semi-supervised, and unsupervised (blind) linear unmixing. Advances in image processing and machine learning substantially affected unmixing. This paper provides an overview of advanced and conventional unmixing approaches. Additionally, we draw a critical comparison between advanced and conventional techniques from the three categories. We compare the performance of the unmixing techniques on three simulated and one real dataset. The experimental results reveal the advantages of different unmixing categories for different unmixing scenarios. Moreover, we provide an open-source Python-based package available at https://github.com/BehnoodRasti/HySUPP to reproduce the results.

In this article, we introduce a novel linear model tailored for semisupervised/library-based unmixing. Our model incorporates considerations for library mismatch while enabling the enforcement of the abundance sum-to-one constraint (ASC). Unlike conventional sparse unmixing methods, this model involves nonconvex optimization, presenting significant computational challenges. We demonstrate the efficacy of alternating direction method of multipliers (ADMM) in cyclically solving these intricate problems. We propose two semisupervised unmixing approaches, each relying on distinct priors applied to the new model in addition to the ASC: sparsity prior and convexity constraint. Our experimental results validate that enforcing the convexity constraint outperforms the sparsity prior for the endmember library. These results are corroborated across three simulated datasets (accounting for spectral variability and varying pixel purity levels) and the Cuprite dataset. In addition, our comparison with conventional sparse unmixing methods showcases considerable advantages of our proposed model, which entails nonconvex optimization. Notably, our implementations of the proposed algorithms—fast semisupervised unmixing (FaSUn) and sparse unmixing using soft shrinkage (SUnS)—prove considerably more efficient than traditional sparse unmixing methods. SUnS and FaSUn were implemented using PyTorch and provided in a dedicated Python package called FaSUn, which is open-source and available athttps://github.com/BehnoodRasti/FUnmix.

Since many materials behave as heterogeneous intimate mixtures with which each photon interacts differently, the relationship between spectral reflectance and material composition is very complex. Quantitative validation of spectral unmixing algorithms requires high-quality ground truth fractional abundance data, which are very difficult to obtain.In this work, we generated a comprehensive hyperspectral dataset of intimate mineral powder mixtures by homogeneously mixing five different clay powders (Kaolin, Roof clay, Red clay, mixed clay, and Calcium hydroxide). In total 325 samples were prepared. Among the 325 samples, 60 mixtures were binary, 150 were ternary, 100 were quaternary, and 15 were quinary. For each mixture (and pure clay powder), reflectance spectra are acquired by 13 different sensors, with a broad wavelength range between the visible and the long-wavelength infrared regions (i.e., between 350 nm and 15385 nm) and with a large variation in sensor types, platforms, and acquisition conditions. We will make this dataset public, to be used by the community for the validation of nonlinear unmixing methodologies (https://github.com/VisionlabUA/Multisensor_datasets)

Hyperspectral datasets obtained from a specific sensor can experience changes in their characteristics due to environmental noise and variations in illumination. Consequently, a segmentation model trained on one dataset may struggle to accurately predict labels and detect objects on a different dataset due to discrepancies between the two domains. To overcome this challenge, domain adaptation techniques can be employed. In the paper, we study hyperspectral domain adaptation for adapting the target domain to align with the source domain in detecting the material type of mm-scale particles from shredded electrolyzers on a conveyor belt for recycling applications. This is necessary due to the non-uniform distribution of particles, variations in material types, and changes in the imaging environment. The results show improvements compared to a pre-trained model using a 2D convolutional neural network.

This paper introduces an open-source hyperspectral unmixing Python package called HySUPP. Hyperspectral unmixing can be divided into three main categories considering the prior knowledge of the endmembers; supervised, semi-supervised, and unsupervised (blind) unmixing. HySUPP includes more than 20 Python-based unmixing approaches from different categories. In the experimental section, we use a few candidates of every category and compare different unmixing approaches. In the experimental section, a challenging simulated dataset without pure pixels is used. The results confirm the advantages of deep learning-based unmixing approaches compared to conventional techniques for all categories in terms of abundance root mean square error. Additionally, blind unmixing approaches outperform supervised and semi-supervised unmixing. The package can be found at: https://github.com/BehnoodRasti/HySUPP.

This letter introduces a new sparse unmixing technique using archetypal analysis (SUnAA). First, we design a new model based on archetypal analysis (AA). We assume that the endmembers of interest are a convex combination of endmembers provided by a spectral library and that the number of endmembers of interest is known. Then, we propose a minimization problem. Unlike most conventional sparse unmixing methods, here the minimization problem is nonconvex. We minimize the optimization objective iteratively using an active set algorithm. Our method is robust to the initialization and only requires the number of endmembers of interest. SUnAA is evaluated using two simulated datasets for which results confirm its better performance over other conventional and advanced techniques in terms of signal-to-reconstruction error (SRE). SUnAA is also applied to Cuprite dataset and the results are compared visually with the available geological map provided for this dataset. The qualitative assessment demonstrates the successful estimation of the minerals abundances and significantly improves the detection of dominant minerals compared to the conventional regression-based sparse unmixing methods. The Python implementation of SUnAA can be found at:https://github.com/BehnoodRasti/SUnAA.

Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a distinctive representation in a reduced and hierarchical feature space. Extensive experiments are carried out on widely used benchmark datasets i.e., the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at https://github.com/AnkurDeria/MFT.

In this paper, we introduce a new algorithm based on archetypal analysis for blind hyperspectral unmixing, assuming linear mixing of endmembers. Archetypal analysis is a natural formulation for this task. This method does not require the presence of pure pixels (i.e., pixels containing a single material) but instead represents endmembers as convex combinations of a few pixels present in the original hyperspectral image. Our approach leverages an entropic gradient descent strategy, which (i) provides better solutions for hyperspectral unmixing than traditional archetypal analysis algorithms, and (ii) leads to efficient GPU implementations. Since running a single instance of our algorithm is fast, we also propose an ensembling mechanism along with an appropriate model selection procedure that make our method robust to hyper-parameter choices while keeping the computational complexity reasonable. By using six standard real datasets, we show that our approach outperforms state-of-the-art matrix factorization and recent deep learning methods. We also provide an open-source PyTorch implementation: https://github.com/inria-thoth/EDAA.

In this paper, we propose a deep learning-based hyperspectral inpainting (DeepHyIn). The proposed approach is unsupervised since it only utilizes the observed image for training the network. First, we propose a novel model for hyperspectral inpainting in which the degraded hyperspectral image is a linear mixture of endmembers and degraded abundances. The proposed model is subjected to abundance sum to one and nonnegativity constraints. We further assume that the endmembers are known. Then, we propose an optimization problem to estimate the unknown abundance using an image prior. Inspired by deep image prior, we shift the optimization problem to optimize the parameters of a deep network. The proposed method uses a deep convolutional encoder-decoder architecture as a backbone. Finally, we apply the DeepHyIn to the Samson dataset and evaluate the results. DeepHyIn demonstrates considerable quantitative and qualitative improvements compared with the state-of-the-art. DeepHyIn was implemented in Python (3.9) using PyTorch as the platform for the deep network and is available online: https://github.com/BehnoodRasti/DeepHyIn.

This paper proposes a sparse unmixing technique using a convolutional neural network (SUnCNN). We reformulate the sparse unmixing problem into an optimization over the parameters of a convolutional network. Relying on a spectral library, the deep network learns in an unsuper-vised manner a mapping from a fixed input to the sparse abundances. Moreover, SUnCNN fulfills the sum-to-one constraint using a softmax activation layer. We compare SUnCNN with the state-of-the-art using a simulated and a real dataset. The experimental results show that the proposed deep learning-based unmixing method outperforms the oth-ers in terms of signal to reconstruction error. Additionally, SUnCNN is visually superior to the competing techniques. SUnCNN was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online: https://github.com/BehnoodRasti/SUnCNN.

This paper proposes a deep blind hyperspectral unmixing network for datasets without pure pixels called minimum simplex convolutional network (MiSiCNet). MiSiCNet is the first deep learning-based blind unmixing method proposed in the literature which incorporates both spatial and geometrical information of the hyperspectral data, in addition to the spectral information. The proposed convolutional encoder-decoder architecture incorporates the spatial information using convolutional filters and implicitly applying a prior on the abundances. We added a minimum simplex volume penalty term to the loss function to exploit the geometrical information. We evaluate the performance of MiSiCNet on simulated and real datasets. The experimental results confirm the robustness of the proposed method to both noise and absence of pure pixels. Additionally, MiSiCNet considerably outperforms the state-of-the-art unmixing approaches. The results are given in terms of spectral angle distance in degree for the endmember estimation, and root mean square error in percentage for the abundance estimation. MiS-iCNet was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online: https://github.com/BehnoodRasti/MiSiCNet.

Hyperspectral imaging is an important technology in the field of geosciences and remote sensing.However, the highdimensional nature of hyperspectral images (HSIs) together with the limited availability of training/labeled samples challenge an efficient processing of HSIs.To alleviate these challenges, we propose a deep multi-resolution clustering network (DMC-Net) to analyze HSIs.DMC-Net, without requiring training/labeled samples for the training process, captures the non-linear intrinsic relation within data points in an HSI and analyzes the image at various resolutions by applying atrous convolutions.Furthermore, DMC-Net preserves the spectral information by directly incorporating extracted features from the original HSI into the reconstruction phase.In terms of clustering accuracy, experimental results on two real HSIs demonstrate the superior performance of DMC-Net compared to the state-of-the-art deep learning-based clustering approaches.

We propose a new fusion-based classification technique for optical multisource remote-sensing images called OptFus. OptFus is developed to merge and process optical imagery having different spatial and spectral resolutions. The spatial features are extracted using morphological filters from the RGB data containing high spatial resolution. A feature fusion technique is developed to combine all the sensor data in a subspace using a common set of representative features. Finally, the fused features are classified using a support vector machine to ensure a robust supervised spectral classification. The proposed method is designed to allocate varying weights to the data from various imaging sensors in the fusion process. OptFus is applied to two multisource optical datasets captured from geological drill-core samples. The classification accuracy demonstrates considerable improvements compared to the state-of-the-art. A MATLAB implementation of OptFus is available online:https://github.com/BehnoodRasti/OptFus.

In this letter, we propose a sparse unmixing technique using a convolutional neural network (SUnCNN) for hyperspectral images. SUnCNN is the first deep learning-based technique proposed for sparse unmixing. It uses a deep convolutional encoder–decoder to generate the abundances relying on a spectral library. We reformulate the sparse unmixing into an optimization over the deep network’s parameters. Therefore, the deep network learns in an unsupervised manner to map a fixed input into the sparse optimum abundances. Additionally, SUnCNN holds the sum-to-one constraint using a softmax activation layer. The proposed method is compared with the state-of-the-art using two synthetic datasets and one real hyperspectral dataset. The overall results confirm that the proposed method outperforms the other ones in terms of signal to reconstruction error (SRE). Additionally, SUnCNN shows visual superiority for both real and synthetic datasets compared with the competing techniques. The proposed method was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online:https://github.com/BehnoodRasti/SUnCNN.

Transformers have intrigued the vision research community with their state-of-the-art performance in natural language processing. With their superior performance, transformers have found their way in the field of hyperspectral image classification and achieved promising results. In this article, we harness the power of transformers to conquer the task of hyperspectral unmixing and propose a novel deep neural network-based unmixing model with transformers. A transformer network captures nonlocal feature dependencies by interactions between image patches, which are not employed in CNN models, and hereby has the ability to enhance the quality of the endmember spectra and the abundance maps. The proposed model is a combination of a convolutional autoencoder and a transformer. The hyperspectral data is encoded by the convolutional encoder. The transformer captures long-range dependencies between the representations derived from the encoder. The data are reconstructed using a convolutional decoder. We applied the proposed unmixing model to three widely used unmixing datasets, i.e., Samson, Apex, and Washington DC mall and compared it with the state-of-the-art in terms of root mean squared error and spectral angle distance. The source code for the proposed model will be made publicly available at https://github.com/preetam22n/DeepTrans-HSU.

This paper proposes a blind nonlinear unmixing technique for intimate mixtures using the Hapke model and convolutional neural networks (HapkeCNN). We use the Hapke model and a fully convolutional encoder-decoder deep network for the nonlinear unmixing. Additionally, we propose a novel loss function that includes three terms; 1) a quadratic term based on the Hapke model, that captures the nonlinearity, 2) the reconstruction error of the reflectances, to ensure the fidelity of the reconstructed reflectance, and 3) a minimum volume total variation term that exploits the geometrical information to estimate the endmembers in the absence of pure pixels in the hyperspectral data. The proposed method is evaluated using two simulated and two real datasets. We compare the results of endmember and abundance estimation with a number of nonlinear, and projection-based linear unmixing techniques. The experimental results confirm that HapkeCNN considerably outperforms the state-of-the-art nonlinear approaches.The proposed method was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available at: https://github.com/BehnoodRasti/HapkeCNN.

In this article, we propose a minimum simplex convolutional network (MiSiCNet) for deep hyperspectral unmixing. Unlike all the deep learning-based unmixing methods proposed in the literature, the proposed convolutional encoder–decoder architecture incorporates spatial information and geometrical information of the hyperspectral data in addition to the spectral information. The spatial information is incorporated using convolutional filters and implicitly applying a prior on the abundances. The geometrical information is exploited by incorporating a minimum simplex volume penalty term in the loss function for the endmember estimation. This term is beneficial when there are no pure material pixels in the data, which is often the case in real-world applications. We generated simulated datasets, where we consider two different no-pure pixel scenarios. In the first scenario, there are no pure pixels but at least two pixels on each facet of the data simplex (i.e., mixtures of two pure materials). The second scenario is a complex case with no pure pixels and only one pixel on each facet of the data simplex. In addition, we evaluate the performance of MiSiCNet in three real datasets. The experimental results confirm the robustness of the proposed method to both noise and the absence of pure pixels. In addition, MiSiCNet considerably outperforms the state-of-the-art unmixing approaches. The results are given in terms of spectral angle distance in degree for the endmember estimation and the root mean square error in percentage for the abundance estimation. MiSiCNet was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online:https://github.com/BehnoodRasti/MiSiCNet.

In this article, we introduce a deep learning-based technique for the linear hyperspectral unmixing problem. The proposed method contains two main steps. First, the endmembers are extracted using a geometric endmember extraction method, i.e., a simplex volume maximization in the subspace of the data set. Then, the abundances are estimated using a deep image prior. The main motivation of this work is to boost the abundance estimation and make the unmixing problem robust to noise. The proposed deep image prior uses a convolutional neural network to estimate the fractional abundances, relying on the extracted endmembers and the observed hyperspectral data set. The proposed method is evaluated on simulated and three real remote sensing data for a range of SNR values (i.e., from 20 to 50 dB). The results show considerable improvements compared to state-of-the-art methods. The proposed method was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online:https://github.com/BehnoodRasti/UnDIP.

In this paper, we introduce ‘Unmixing Deep Image Prior’ (UnDIP), a deep learning-based technique for the linear hyperspectral unmixing problem. The proposed method contains two steps. First, the endmembers are extracted using a geometric endmember extraction method, i.e. a simplex volume maximization in a subspace of the dataset. Then, the abundances are estimated using a deep image prior. The proposed deep image prior uses a convolutional neural network to estimate the fractional abundances, relying on the extracted endmembers and the observed hyperspectral dataset. The results show considerable improvements compared to state-of-the-art methods.

We present an analysis of the influence of noise on the unmixing of hyperspectral data. We propose four scenarios to 1) investigate the effect of noise reduction as a preprocessing step on the performance of hyperspectral unmixing and 2) study the relation between noise and different endmembers selection strategies. Experiments are conducted on a simu-1ated and a real datasets with a wide range of signal to noise ratios (from 10 to 50 dB).

Remote sensing data is contaminated with different types of noise that can severely affect the analysis of this data. Generally, in modern treatment chains of satellite and aerial data, denoising techniques are applied to atmospherically corrected images prior to further analysis (e.g., classification). However, since the noise contaminates the measured radiance at the sensor, it can influence the atmospheric correction in itself and consequently the remaining of the processing chain. In this paper, we compare the performance of a denoising technique, when applied before or after atmospheric correction. Our observations challenge the current de facto paradigm of denoising in a processing chain of spaceborne and airborne remotely sensed images.

The inclusion of spatial information into spectral classifiers for fine-resolution hyperspectral imagery has led to significant improvements in terms of classification performance. The task of spectral-spatial hyperspectral image (HSI) classification has remained challenging because of high intraclass spectrum variability and low interclass spectral variability. This fact has made the extraction of spatial information highly active. In this work, a novel HSI classification framework using the fusion of dual spatial information is proposed, in which the dual spatial information is built by both exploiting pre-processing feature extraction and post-processing spatial optimization. In the feature extraction stage, an adaptive texture smoothing method is proposed to construct the structural profile (SP), which makes it possible to precisely extract discriminative features from HSIs. The SP extraction method is used here for the first time in the remote sensing community. Then, the extracted SP is fed into a spectral classifier. In the spatial optimization stage, a pixel-level classifier is used to obtain the class probability followed by an extended random walker-based spatial optimization technique. Finally, a decision fusion rule is utilized to fuse the class probabilities obtained by the two different stages. Experiments performed on three data sets from different scenes illustrate that the proposed method can outperform other state-of-the-art classification techniques. In addition, the proposed feature extraction method, i.e., SP, can effectively improve the discrimination between different land covers.

This paper brings together advances in remote sensing and deep learning for mineral mapping in a sustainable way. In more detail, we propose a multisensor feature fusion approach to integrate heterogeneous RGB, multispectral, and hyperspectral images for sustainable mining. The proposed approach is composed of two main steps; Feature extraction and classification. In the feature extraction step, we develop a three-stream convolutional neural network to extract high-level information from the input multisensor data. In the classification step, we develop a multisensor composite kernel approach to perform fusion and mapping simultaneously. The proposed approach produces very high quality classification maps with exceptional results in terms of classification accuracies.

Accurately mapping lithology and geological structures remains a challenge in rough terrain or in active mining areas. We propose that the integration of terrestrial and drone-borne multi-sensor remote sensing techniques can significantly boost the reliability, safety, and efficiency of geological activities in exploration and for the monitoring of mining activities. We have now developed a complete procedural chain to jointly and accurately process Structure-from-Motion Multi-View Stereo point clouds and hyperspectral data cubes in the visible to near-infrared (VNIR) and short-wave infrared (SWIR), as well as long-wave infrared (LWIR) ranges acquired by terrestrial sensors. Hyperspectral data are processed using spectroscopic and machine learning algorithms to generate meaningful 2.5D (i.e., surface) maps that are available to geologists on the ground shortly after data acquisition. We classify the geological information content using innovative machine learning techniques. We validate the remote sensing data with in-situ mineralogical and structural measurements. Repeated acquisitions allow then to integrate a time component.

This paper presents a technique for the fusion of multispectral LiDAR and hyperspectral data. The proposed method is based on the fusion of the features of multispectral LiDAR and hyperspectral data projected in two different subspaces. First, the spatial features are extracted from both data using morphological filters. Then, the fused features are estimated by proposing a novel constraint penalized cost function. The estimated fused features are used for the purpose of mapping. The classification accuracies obtained by applying a random forest classifier on the fused data confirm considerable improvements compared with the other methods used in the experiments.

Hyperspectral images (HSIs) are often degraded by different noise types such as Gaussian and sparse noise. In this letter, a hyperspectral mixed Gaussian and sparse noise reduction technique, the HyMiNoR, is proposed. The proposed technique, hierarchically, removes the mixed noise. First, the Gaussian noise is removed using a recently developed automatic hyperspectral noise removal technique called hyperspectral restoration (HyRes). Then, we develop a novel sparse noise removal technique to remove the sparse noise, including salt and pepper noise, missing pixels, and missing lines. The performance of the proposed approach has been validated using both real and simulated data sets. Results on the simulated data set confirm considerable improvements in terms of signal-to-noise ratio and singular angle distance compared to the state-of-the-art techniques used in the experiments. In addition, visual improvements can be clearly observed in the case of real data set experiments.

The increased availability of active and passive data captured over the same scene of interest makes it desirable to jointly utilize multisensor data to perform accurate classification. This paper proposes a novel fusion approach to integrate hyperspectral and LiDAR-derived digital surface model for land-cover classification. In this context, we propose a novel multisensor composite kernel technique based on extreme learning machines (named as multisensor composite kernels (MCKs)), which is capable of combining different methods in the feature fusion level in an effective way. In the proposed approach, we use extinction profiles to extract spatial and elevation features of hyperspectral and LiDAR data. Then, hyperspectral Stein's unbiased risk estimator (HySURE) is applied to identify the subspace (informative features) of spectral, spatial, and elevation features. Finally, MCK is applied to the extracted spectral, spatial, and elevation features to produce the final classification map. Results obtained by the proposed approach reveal the fact that this approach can effectively fuse and classify hyperspectral and LiDAR images and improve the classification accuracy of each data source significantly. In addition, the proposed method is fully automatic.

In this paper, we propose a framework to fuse features extracted from hyperspectral and Light Detection And Ranging (LiDAR)-derived data. Spatial and elevation features are extracted from multisensor data using extinction profiles (EP). All the features, including the spectral ones, are fused using sparse and smooth low-rank analysis (SSLRA). In terms of classification accuracy, the proposed framework outperforms other studied fusion techniques used in the experiments.

In this letter, we first propose multisensor composite kernel (MCK) extreme learning machines to fuse hyperspectral and light detection and ranging (LiDAR) features effectively. Then, based on the MCK, we develop a fully automatic fusion framework. In the proposed framework, spatial and elevation features of hyperspectral and LiDAR data are first extracted using extinction profiles. Then, hyperspectral Stein's unbiased risk estimator is utilized to extract the subspace (informative features) of spectral, spatial, and elevation features. The obtained results indicate that the proposed approach can successfully integrate and classify hyperspectral and LiDAR images to provide accurate classification results classification accuracies in an automatic manner.

In this paper, a hyperspectral feature extraction (FE) method called sparse and smooth low-rank analysis (SSLRA) is proposed. First, we propose a new low-rank model for hyperspectral images (HSIs). In the new model, HSI is decomposed into smooth and sparse unknown features which live in an unknown orthogonal subspace. Then, the sparse and smooth features are simultaneously estimated using a non-convex constrained penalized cost function. In the experiments' SSLRA is applied on a real HSI and the smooth features extracted are used for the HSI classification. The results confirm improvements in classification accuracies compared to state-of-the-art FE methods.

To improve the classification of hyperspectral images, this paper proposes an approach for multi-sensor data fusion of LiDAR and hyperspectral data using extinction profiles and Orthogonal Total Variation Component Analysis (OTVCA). Results on the benchmark Houston data indicate the superior performance of the proposed approach compared to other approaches used in the experiments based on classification accuracies.

Hyperspectral restoration is a preprocessing step for hyperspectral imagery. In this letter, we propose a parameter-free method for the restoration of hyperspectral images (HSIs) called HyRes. The restoration method is based on a sparse low-rank model that uses the ℓ1penalized least squares for estimating the unknown signal. The Stein's unbiased risk estimator is exploited to select all the parameters of the model yielding a fully automatic (parameter free) technique. Experimental results confirm that HyRes outperforms the state-of-the-art techniques in terms of signal-to-noise ratio, structural similarity index, and spectral angle distance for a simulated data set and in terms of noise-level estimation for the real data sets used in this letter. In the experiments, it was noted that HyRes is computationally less expensive compared with competitive techniques. Therefore, HyRes can be used as a reliable automatic preprocessing step for further analysis of HSIs.

The classification accuracy of remote sensing data can be increased by integrating ancillary data provided by multisource acquisition of the same scene. We propose to merge the spectral and spatial content of hyperspectral images (HSIs) with elevation information from light detection and ranging (LiDAR) measurements. In this paper, we propose to fuse the data sets using orthogonal total variation component analysis (OTVCA). Extinction profiles are used to automatically extract spatial and elevation information from HSI and rasterized LiDAR features. The extracted spatial and elevation information is then fused with spectral information using the OTVCA-based feature fusion method to produce the final classification map. The extracted features have high dimension, and therefore OTVCA estimates the fused features in a lower dimensional space. OTVCA also promotes piece-wise smoothness while maintaining the spatial structures. Both attributes are important to provide homogeneous regions in the final classification maps. We benchmark the proposed approach (OTVCA-fusion) with an urban data set captured over an urban area in Houston/USA and a rural region acquired in Trento/Italy. In the experiments, OTVCA-fusion is evaluated using random forest and support vector machine classifiers. Our experiments demonstrate the ability of OTVCA-fusion to produce accurate classification maps while using fewer features compared with other approaches investigated in this paper.

The availability of diverse data captured over the same region makes it possible to develop multisensor data fusion techniques to further improve the discrimination ability of classifiers. In this paper, a new sparse and low-rank technique is proposed for the fusion of hyperspectral and light detection and ranging (LiDAR)-derived features. The proposed fusion technique consists of two main steps. First, extinction profiles are used to extract spatial and elevation information from hyperspectral and LiDAR data, respectively. Then, the sparse and low-rank technique is utilized to estimate the low-rank fused features from the extracted ones that are eventually used to produce a final classification map. The proposed approach is evaluated over an urban data set captured over Houston, USA, and a rural one captured over Trento, Italy. Experimental results confirm that the proposed fusion technique outperforms the other techniques used in the experiments based on the classification accuracies obtained by random forest and support vector machine classifiers. Moreover, the proposed approach can effectively classify joint LiDAR and hyperspectral data in an ill-posed situation when only a limited number of training samples are available.

In this paper, a novel feature extraction method, called orthogonal total variation component analysis (OTVCA), is proposed for remotely sensed hyperspectral data. The features are extracted by minimizing a total variation (TV) penalized optimization problem. The TV penalty promotes piecewise smoothness of the extracted features which is useful for classification. A cyclic descent algorithm called OTVCA-CD is proposed for solving the minimization problem. In the experiments, OTVCA is applied on a rural hyperspectral image having low spatial resolution and an urban hyperspectral image having high spatial resolution. The features extracted by OTVCA show considerable improvements in terms of classification accuracy compared with features extracted by other state-of-the-art methods.

The identification of the signal subspace is a very important first step for most hyperspectral algorithms. In this letter, we investigate the important problem of identifying the hyperspectral signal subspace by minimizing the mean squared error (MSE) between the true signal and an estimate of the signal. Since it is dependent on the true signal, the MSE is uncomputable in practice, and so we propose a method based on Stein's unbiased risk estimator that provides an unbiased estimate of the MSE. The resulting method is simple and fully automatic, and we evaluate it using both simulated and real hyperspectral data sets. Experimental results show that our proposed method compares well to recent state-of-the-art subspace identification methods.

In this paper, a hyperspectral feature extraction method is proposed. A low-rank linear model using the right eigenvector of the observed data is given for hyperspectral images. A total variation (TV) based regularization called Low-Rank TV regularization (LRTV) is used for hyperspectral feature extraction. The feature extraction is used for hyperspectral image classification. The classification accuracies obtained are significantly better than the ones obtained using features extracted by Principal Component Analysis (PCA) and Maximum Noise Fraction (MNF).

Mean squared error (MSE) is commonly used for evaluating the performance of hyperspectral imaging (HSI) methods. MSE depends on the true (unknown) signal to be estimated and is therefore not computable for real data. Therefore, HSI methods are usually evaluated using simulated data. Stein's unbiased risk estimator (SURE) is an unbiased estimator of the MSE that does not require knowledge of the true signal. The main aim of this paper is to promote the use of SURE for evaluating HSI models. To achieve that goal we compare three wavelet models, spectral, spatial and spectral-spatial, for hyperspectral images. Hyperspectral images are modeled based on their sparse wavelet components. The penalized least squares with i.e. penalty (to promote sparsity) is considered for sparse reconstruction. By comparing the SURE values for the three models, it is shown that the spatial model performs better than spectral model and spectral-spatial model outperforms both spectral and spatial models.

In this paper, a method called wavelet-based sparse reduced-rank regression (WSRRR) is proposed for hyperspectral image restoration. The method is based on minimizing a sparse regularization problem subject to an orthogonality constraint. A cyclic descent-type algorithm is derived for solving the minimization problem. For selecting the tuning parameters, we propose a method based on Stein's unbiased risk estimation. It is shown that the hyperspectral image can be restored using a few sparse components. The method is evaluated using signal-to-noise ratio and spectral angle distance for a simulated noisy data set and by classification accuracies for a real data set. Two different classifiers, namely, support vector machines and random forest, are used in this paper. The method is compared to other restoration methods, and it is shown that WSRRR outperforms them for the simulated noisy data set. It is also shown in the experiments on a real data set that WSRRR not only effectively removes noise but also maintains more fine features compared to other methods used. WSRRR also gives higher classification accuracies.

This paper deals with hyperspectral image reconstruction using a new linear model and Sparse Regularization (SR). The new model is based on Principal Components (PCs) and wavelets. Since the hyperspectral PCs are not spatially sparse, wavelet is applied to get spatially sparse representation. Sparse regularization is used to recover the corrupted signal. The regularization parameter is chosen by Stein's Unbiased Risk Estimator (SURE). The results show improvements for simulated data sets compare to other denoising methods based on Signal to Noise Ratio (SNR). In addition, the methods are applied on a real noisy data set, and the results of the new method demonstrate visual improvement. The proposed approach is automatic, fast and has the ability to be applied on very large data sets.

In this paper, we propose a denoising method for hyperspectral images using 3D wavelets. We use the sparse analysis regularization using a 3D overcomplete wavelet dictionary. The minimization problem is solved using iterative Chambolle algorithm. The simulation results show that the 3D dictionary outperforms the 2D one, in terms of Peak Signal to Noise Ratio (PSNR). Denosing hysperspectral cubes is likely to increase the classification accuracy of the hyperspectral data since it can enhance the spectral profiles (or features) that can be useful to discriminate between information classes.