Hyperspectral change detection (HCD) has found numerous key applications, such as land cover monitoring. The majority of benchmark HCD algorithms are semi-supervised methods, and some of them can even achieve very low sample labeling rates. However, in some practical scenarios, such as those requiring immediate detection responses for onboard edge computing, we need to achieve the zero-label requirement as ground-truth labeling would not be available onboard for newly acquired images. In this work, we propose a fully unsupervised HCD algorithm, together with a lightweight model, quite suitable for onboard detection missions. Based on an iteratively augmented training set that safely collects some unchanged pixel samples, we learn an iteratively refined spectrum calibration function that eventually compensates the variability of acquisition conditions (often observed in bitemporal images), thereby making the changed pixels easily detectable by analyzing the calibrated spectra. The proposed hyperspectral looping unsupervised calibration and incremental detection (HyperLUCID) algorithm is not only computationally efficient (around 1 to 2 orders of magnitude faster than most benchmark HCD methods), but has also achieved state-of-the-art results (around 93.6% to 97.9% overall accuracy) on several real benchmark HCD datasets. Source codes: https://github.com/IHCLab/HyperLUCID.

SAR-to-optical translator networks are especially used to overcome the lack of optical images under cloudy conditions. Those translations being used for downstream tasks, they require the reconstruction of reliable patterns with respect to the underlying objects. In this paper, we propose a novel training strategy to account for land-cover complexity through a conjoint Translation-Classification Loss (TCL). The proposed loss evaluates the classifiability of translated images with a pre-trained land-cover classifier by assessing the reliability of its predictions and the relevance of its extracted hidden features. This new loss is applied to nine translators from the literature and to a tenth architecture introduced in the paper. Experiments show that applying the TCL not only improves the credibility of structures, patterns and textures but it also allows for better class discrimination and transitions while avoiding unreliable hallucinated artifacts produced by standard losses in adversarial approaches.

SAR colorization aims to enrich gray-scale SAR images with color while ensuring the preservation of original radiometric and spatial details. However, researchers often limit themselves to using only the red, green, and blue bands of a multispectral image as the source of color information, coupled with a single-polarization channel from the SAR image. This approach neglects the intrinsic characteristics of remote sensing data and thus fails to fully leverage available information. To overcome this limitation, this research attempts to explore inclusion of all available bands from multispectral images along with dual-polarization channels from SAR imagery in the colorization process. Furthermore, we present a new colorization method called improved conditional generative adversarial network for SAR colorization (IcGAN4ColSAR). This method tries to include the spectral angle mapper index within its loss function. Sufficient experiments show that our explorations in the number of data channels and the loss function are helpful in improving the colorization performance of the SAR image.

Multi-source image fusion combines the information coming from multiple images into one data, thus improving imaging quality. This topic has aroused great interest in the community. How to integrate information from different sources is still a big challenge, although the existing self-attention based transformer methods can capture spatial and channel similarities. In this paper, we first discuss the mathematical concepts behind the proposed generalized self-attention mechanism, where the existing self-attentions are considered basic forms. The proposed mechanism employs multilinear algebra to drive the development of a novel fully-connected self-attention (FCSA) method to fully exploit local and non-local domain-specific correlations among multi-source images. Moreover, we propose a multi-source image representation embedding it into the FCSA framework as a non-local prior within an optimization problem. Some different fusion problems are unfolded into the proposed fully-connected transformer fusion network (FC-Former). More specifically, the concept of generalized self-attention can promote the potential development of self-attention. Hence, the FC-Former can be viewed as a network model unifying different fusion tasks. Compared with state-of-the-art methods, the proposed FC-Former method exhibits robust and superior performance, showing its capability of faithfully preserving information.

Multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution.

Multi-modal image fusion aims to generate a fused image by integrating and distinguishing the cross-modality complementary information from multiple source images. While the cross-attention mechanism with global spatial interactions appears promising, it only captures second-order spatial interactions, neglecting higher-order interactions in both spatial and channel dimensions. This limitation hampers the exploitation of synergies between multi-modalities. To bridge this gap, we introduce a Synergistic High-order Interaction Paradigm (SHIP), designed to systematically investigate spatial fine-grained and global statistics collaborations between the multi-modal images across two fundamental dimensions: 1) Spatial dimension: we construct spatial fine-grained interactions through element-wise multiplication, mathematically equivalent to global interactions, and then foster high-order formats by iteratively aggregating and evolving complementary information, enhancing both efficiency and flexibility. 2) Channel dimension: expanding on channel interactions with first-order statistics (mean), we devise high-order channel interactions to facilitate the discernment of inter-dependencies between source images based on global statistics. We further introduce an enhanced version of the SHIP model, called SHIP++ that enhances the cross-modality information interaction representation by the cross-order attention evolving mechanism, cross-order information integration, and residual information memorizing mechanism. Harnessing high-order interactions significantly enhances our model's ability to exploit multi-modal synergies, leading in superior performance over state-of-the-art alternatives, as shown through comprehensive experiments across various benchmarks in two significant multi-modal image fusion tasks: pan-sharpening, and infrared and visible image fusion.

Detecting slope instabilities on Synthetic Aperture Radar (SAR) interferograms using deep learning approaches presents several challenges. This detection task suffers from the lack of transparency of deep networks, the complexity of the input data (i.e. complex values, sensitivity to distortions and presence of counterfactuals) and the complexity of the target phenomena (i.e. the variable velocities and the complex underground processes). In this paper, we propose a new framework called ”Explainability Constrained-claSsifier for Pairing the detection and the Localization of moving Areas on INterferograms” (ECSPLAIN), to generate decision, localization and segmentation maps from a single but explainable classifier network. It consists of training a classifier to detect whether an instability is located in the patch or not, and to explain its decision with a Class Activation Map (CAM) that matches the actual location of the instability. Therefore by using a single classifier network, the framework can pair the detection and the localization of moving areas. Four CAMs are investigated for the training of the ECSPLAIN framework. Experiments on the ISSLIDE dataset show that our proposal achieves better explainability than standarda posterioriCAMs with more than 0.20 points of improvements in terms of Dice and IoU scores. It also allows competitive performance with segmentation-only networks with only 0.04 points of difference in terms of Dice and IoU scores. Thus, the proposed method is competitive with the most efficient methods while being lighter, faster, and delivering a decision based on a human-like reasoning process. Finally, the ECSPLAIN framework is applied to enrich the ISSLIDE dataset, discovering more than 470 manually validated slope instabilities over the Alps.

Nowadays, various graph convolutional networks (GCNs) to process graph-structured data have been proposed for hyperspectral image (HSI) classification. Nevertheless, most GCN-based HSI classification methods emphasize graph node feature aggregation instead of graph pooling, resulting in them being shallow networks and unable to extract deep discriminative features. Besides, to obtain the new graph after the pooling layer, current graph pooling methods used for HSI classification just consider node feature information to select important nodes and directly discard unselected nodes, which could be a subjective process and may cause information loss. To solve this issue, we propose a novel graph U-Net with topology-feature awareness pooling (the so-called TFAP graph U-Net) for HSI classification considering a deep network to extract compelling features and automatically selecting nodes beneficial to classification. More specifically, to establish a more precise pooled graph, the graph’s topology structure and node feature information are taken into account, making the node selection process more convincing and objective. Furthermore, to allow that graph nodes preserve more useful graph information, our method aggregates node features from neighboring nodes that may not be selected, which can alleviate the loss of information during the pooling process. Moreover, a cross-attention module (CAM) is used to filter out irrelevant or noisy features. Finally, we evaluate the proposed method on three public HSI datasets, i.e., Indian Pines, University of Pavia (PaviaU), and University of Houston. The experimental results demonstrate the superiority of the proposed approach compared with other state-of-the-art methods.

In this paper, a multimodal deep architecture for classification of light detection and ranging (LiDAR) and hyperspectral image (HSI) is proposed, acquiring the knowledge of both modalities by leveraging modality specific information as well as their complementary information. The proposed model consists of two main steps. First, to improve the performance of a two-dimensional convolutional neural network (2DCNN), low-frequency features with maximum autocorrelation factor of HSI are injected into 2DCNN which are called multi-scale features of 2DCNN. Second, to improve the accuracy of 2DCNN and extract smooth and semantic information, the posterior energy of hidden Markov random field (HMRF) is modified by using Gaussian attention and albedo recovery attention mechanisms and energies of LiDAR and HMRF. Then, these features are fused based on another attention mechanism called attention-based HMRF. Moreover, this HMRF model is used for fusion of HSI and LiDAR. The proposed model is tested on the Houston 2013, Trento and MUUFL datasets and compared with several state-of-the-art methods. The resulting classification accuracies through ablation study show the superior performance of the proposed method.

Deep learning-based frameworks have shown great potential in the field of hyperspectral image (HSI) classification owing to their superior modeling capabilities. However, the existence of mixed pixels and spectral heterogeneity limits the discriminant performance of the classifier, which makes it impossible to distinguish the mixed spectra effectively in actual scenarios. To address this gap, we propose a subpixel spectral variability network ($\text {S}^{2}\text {VNet}$) for HSI classification, which incorporates complete subpixel information and class features modeled by spectral variability and nonlinear mixture characteristics to enhance classification performance.$\text {S}^{2}\text {VNet}$is capable of extracting endmembers and abundances based on the nonlinear autoencoder (AE) framework and estimating variability parameters by simultaneously considering scaling factors and perturbation terms to ensure accurate endmember construction. The enhanced subpixel fusion module is further designed to automatically integrate three aspects of abundances, spectral cosine correlation information, and pixel-level class features to provide a robust joint representation for the classifier. Extensive experiments on four public HSI datasets demonstrate the superiority and generalization of the proposed method when benchmarked with state-of-the-art methods. The code will be available athttps://github.com/hanzhu97702/S2VNet.

Depending on a large-scale paired dataset of low-resolution hyperspectral image (LrHSI), high-resolution multispectral image (HrMSI), and corresponding high-resolution hyperspectral image (HrHSI), the supervised paradigm has achieved impressive performance in the hyperspectral image super-resolution (HISR). However, the intrinsic data-intensive manner hinders its further application in real scenarios. Fortunately, deep image prior (DIP) allows us to achieve unsupervised super-resolution (SR) by solely utilizing degraded observations. However, its potential to accurately model complicated hyperspectral priors is still not fully exploited due to the following two factors: 1) existing methods tend to reconstruct the unknown HrHSI directly from a randomly generated noise, leaving it hard to leverage the scene-relevant information for prior learning and 2) the vanilla architecture is handcrafted for the generator network, which shows limitations in feature representation and thus fails to characterize the complicated image properties. To unleash the potential of DIP for the HISR task, we propose an enhanced DIP network, called EDIP-Net, by addressing the aforementioned impediments. Specifically, EDIP-Net is built with a two-stage four-component scheme, with a zero-shot learning (ZSL) stage for input image establishment and a deep image generation (DIG) stage for prior learning. First, we exploit the cross-scale spectral relationship inside the observations and thus design a degradation learning network to generate paired training samples from the observations themselves. As such, two image-coarse estimations are derived in a ZSL manner by learning an interactive spectral learning network. By replacing random noise with two estimations, we design a double U-shape architecture for the generator network to capture their hyperspectral prior, each independently generating one HrHSI candidate. Under this premise, we further propose a degradation-aware decision fusion strategy to integrate the optimal results in a pixel-to-pixel manner. Extensive experiments demonstrate our superiority in achieving high-quality SR performance. The code will be available athttps://github.com/JiaxinLiCAS.

As the requirements for the downstream tasks of land cover classification (LCC) continue to increase, the category system used for LCC is constantly being refined. This causes previous land cover products and manually annotated training samples to become quickly outdated. Meanwhile, manually annotating samples with more refined categories is extremely time consuming. To address this impasse, a cross-category knowledge transfer process is needed that can directly generate land cover products under a fine-scale category system using existing training samples under a large-scale category system. Accordingly, this paper proposes a cross-category LCC method called conditional Gaussian enhanced dense correlation matching (CGE-DCM). CGE-DCM uses samples under a large-scale category system for training. It then uses only one annotated example of each fine-scale category to achieve fine-scale LCC. In cases with very little sample support, the problems caused by different spectra of the same object and different objects of the same spectrum in complex scenes can be particularly severe. To solve this problem and improve the accuracy of classifications of complex objects, CGE-DCM offers a dense correlation matching strategy. In addition, context distribution is different under fine-scale category systems than it is under large-scale category systems. For this reason, CGE-DCM features a conditional Gaussian enhancement mechanism and designs different loss functions for degenerated and nondegenerated scenarios to ensure the stability of the model. Extensive experiments on Gaofen-2 and Orbita hyperspectral satellite images demonstrate the effectiveness of each module in CGE-DCM and its superiority over existing methods.

Local and global spectral and spatial information is crucial for hyperspectral image (HSI) classification. However, modeling the global context has been challenging due to the limitations of receptive fields and quadratic complexity. Mamba’s ability to leverage long-range dependencies with linear computational complexity offers an effective approach to alleviate this issue; however, it does lead to the loss of local detail information. To address this challenge, we propose a novel local-to-global Mamba for HSI classification, termed MambaLG. MambaLG consists of a dual-branch strategy, comprising two core modules: a local and global spatial modeling module (SpaM) and a short- and long-range spectral dynamic perception module (SpeM). In the SpaM, the local and global spatial information is sequentially extracted and integrated, aiming to capture global spatial semantics while preserving the integrity of local 2-D spatial structures. In the SpeM, we utilize local spectral extraction, spectral grouping, and spectral dynamic correlation clustering (SDCC) modules, leveraging Mamba’s strengths in exploring long-range dependencies for more precise short- and long-range spectral feature modeling. Additionally, we introduce a gate attention unit into MambaLG and design a more efficient and interpretable manner for merging spatial and spectral features. Experimental results across multiple datasets (encompassing urban and agricultural scenes) indicate that MambaLG surpasses state-of-the-art algorithms regarding classification accuracy (CA) and inference speed. Comprehensive ablation studies substantiate the advantages of MambaLG in modeling local and global spatial context, enhancing short- and long-range spectral perception, and fusing spatial and spectral information. The codes will be openly available athttps://github.com/danfenghong/IEEE_TGRS_MambaLGto facilitate the reproduction of experimental results.

Hyperspectral change detection (HCD) techniques to monitor Earth’s surface processes advanced markedly in recent years. Seasonal variations and associated spectral signatures as well as nonlinear noise patterns emanating from sensors and atmospheric sources pose fundamental challenges in HCD. Advanced deep learning models, such as those that leverage convolutional neural networks (3D-Siamese) or transformers (MLP-Mixer), are increasingly employed to address these challenges. However, they often need substantial training data and computational resources. Here, we show that the Kolmogorov–Arnold network (KAN) can enhance HCD capabilities without the excessive training demand of deep networks. The Kolmogorov–Arnold theorem provides the theoretical foundation for our approach, which is particularly well-suited for hyperspectral data analysis by providing a rigorous basis for handling high-dimensional spectral signatures through dimensional reduction and feature extraction. Our architectural design employs this theoretical framework by incorporating specialized neural network layers that mirror the theorem’s compositional structure, thereby facilitating efficient processing of spectral bands. By replacing the linear weighting scheme with learnable nonlinear functions, the Kolmogorov–Arnold network (KAN) provides a unique capability to capture intricate patterns and irregularities in high-dimensional data. Here, we compare five KAN-based architectures and deep learning models such as the MLP-Mixer, 3D-Siamese, dual-branch Siamese spatial–spectral Transformer attention network (DBS3TAN), and the Swin Transformer for HCD and show that the Chebyshev-KAN model, with an average overall accuracy of 97.35% over four real-world benchmark cases, outperforms other models while having a marked lower complexity than the deep learning models. We also show that the choice of fit nonlinear function and model structure is more important than the number of parameters in KAN-based models.

Due to the powerful feature extraction capabilities of deep learning, a series of deep learning-based methods for hyperspectral image (HSI) classification have been proposed and achieved satisfactory performance. However, most of these methods require a large number of labeled data, and the collection of completely accurate pixel-level labeled HSI data is difficult, resulting from the intricate label ambiguity of HSI and incomplete prior knowledge of annotators. Simultaneously, a few researchers focus on label ambiguity for HSI classification. Partial label learning (PLL) is one of the strategies to solve the problem where each training instance is assigned a candidate label set, among which only one is the ground truth label, which can essentially alleviate labeling difficulties. In this article, a max-gap loss-guided Siamese reconstruction network (MAGS) is proposed to combine PLL with HSI classification. MAGS consists of three components, including a spatial-spectral encoder, a spatial-spectral decoder, and a Siamese spatial-spectral encoder for high-quality feature representation learning to facilitate label disambiguation. In the encoding process, MAGS introduces the cross-attention and max-matching fusion strategies to obtain more representative features. In addition, to improve label disambiguation, the maximum gap loss is designed to guide the model training. Quantitative and qualitative results indicate that the MAGS outperforms several state-of-the-art methods on three HSI datasets. The code is available athttps://github.com/Nemo96yu/MAGS.

Displacement estimation using remote sensing images is an effective approach for assessing surface displacement caused by natural disasters like earthquakes and landslides. By employing pixel correlation algorithms, high-precision displacement maps can be generated from images taken before and after surface movement. However, traditional methods often rely on spatial regularization or frequency masking to reduce high-frequency noise, which can smooth spatial details and result in biased displacement estimates, especially near sharp discontinuities typical of earthquake surface ruptures. Moreover, sub-pixel displacement estimation using Synthetic Aperture Radar (SAR) images remains a challenge compared to optical images, due to the strong impact of speckle noise. This paper presents GeoFlowNet-SAR, an innovative sub-pixel displacement estimation method leveraging SAR images. SAR offers advantages thanks to an all-weather observation and high penetration, making it suitable for conditions typically challenging for optical systems in the visible light spectrum. This study uses Sentinel-1 SAR Single Look Complex (SLC) images with dual-polarization (VV and VH modes) and Interferometric Wide (IW) swath mode to balance coverage and resolution. By training on simulated displacement datasets with realistic sharp discontinuities, GeoFlowNet-SAR directly predicts surface displacement fields, providing highly efficient, robust, and precise results, while overcoming some limitations of traditional methods. The effectiveness of the proposed methodological contribution is first quantitatively demonstrated using synthetic simulated earthquake datasets, including comparisons with state-of-the-art correlation methods. The method is further validated using two real remote sensing images from the 2019 Ridgecrest earthquake and from the 2023 Turkey-Syria earthquake. The observed results from these real datasets confirm the effectiveness of GeoFlowNet-SAR in practical applications. The codes are available at: https://gricad-gitlab.univ-grenoble-alpes.fr/giffards/geoflownet-sar.

The joint classification of hyperspectral images (HSIs) and light detection and ranging (LiDAR) data have seen significant advancements in recent research. However, it would be more practical if we could simultaneously detect the unknown classes in a more realistic open-set scenario. In this article, we introduce a novel open-set recognition (OSR) method for HSI and LiDAR data, termed HyLiOSR, which devises a staged progressive learning strategy to effectively bridge the gap between closed-set and open-set feature distributions within an autoencoder framework. Specifically, for the first stage, the reconstruction-based network is dedicated to accurately modeling each known category by learning multiple Gaussian prototypes, which facilitates OSR by disentangling the distribution of known classes. In the second stage, we actively synthesize samples of unknown classes during the feature extraction phase and create a virtual unknown classifier, enabling the network to effectively differentiate between known and unknown class samples. This approach establishes a distinct separation between known and unknown classes in the latent feature space, thereby enhancing the capability of the frameworks to distinguish between them. Comprehensive experiments conducted on three benchmark datasets demonstrate that the proposed HyLiOSR outperforms existing state-of-the-art methods. The source code will be accessible athttps://github.com/B-Xi/TGRS_2025_HyLiOSR.

Hyperspectral image (HSI) classification has been extensively studied in the context of Earth observation. However, its application in Mars exploration remains limited. Although convolutional neural networks (CNNs) have proven effective in HSI processing, their local receptive fields hinder their ability to capture long-range features. Transformers excel in global modeling and perform well in HSI classification (HSIC), but they often neglect the effective representation of local spectral and spatial features and tend to be more complex. To address these challenges, we propose a mixed CNN-transformer network for Mars HSI classification with graph contrastive learning to enhance classification performance. Specifically, we introduce an information-enhanced attention module (IEAM) designed to aggregate attention features from multiple perspectives. Additionally, we develop a lightweight dual-branch CNN-transformer (LDCT) network that efficiently extracts both local and global spectral-spatial features with lower complexity. To improve the discrimination of inter-class features, we apply graph contrastive learning to the topological structure of labeled samples. Furthermore, we annotated three Mars HSI datasets, referred to as HyMars, to validate the effectiveness of our proposed mixed CNN–“transformer network for Mars HSIC with graph contrastive learning (MCTGCL). Comprehensive experimental results across different amounts of labeled samples consistently demonstrate the superiority of the method. The source code is available athttps://github.com/B-Xi/TGRS_2025_MCTGCL.

Few-shot learning (FSL) has been rapidly developed in the hyperspectral image (HSI) classification, potentially eliminating time-consuming and costly labeled data acquisition requirements. Effective feature embedding is empirically significant in FSL methods, which is still challenging for the HSI with rich spectral-spatial information. In addition, compared with inductive FSL, transductive models typically perform better as they explicitly leverage the statistics in the query set. To this end, we devise a transductive FSL framework with enhanced spectral-spatial embedding (TEFSL) to fully exploit the limited prior information available. First, to improve the informative features and suppress the redundant ones contained in the HSI, we devise an attentive feature embedding network (AFEN) comprising a channel calibration module (CCM). Next, a meta-feature interaction module (MFIM) is designed to optimize the support and query features by learning adaptive co-attention using convolutional filters. During inference, we propose an iterative graph-based prototype refinement scheme (iGPRS) to achieve test-time adaptation, making the class centers more representative in a transductive learning manner. Extensive experimental results on four standard benchmarks demonstrate the superiority of our model with various handfuls (i.e., from 1 to 5) labeled samples. The code will be available online at https://github.com/B-Xi/TIP_2025_TEFSL.

Accurately distinguishing between background and anomalous objects within hyperspectral images poses a significant challenge. The primary obstacle lies in the inadequate modeling of prior knowledge, leading to a performance bottleneck in hyperspectral anomaly detection (HAD). In response to this challenge, we put forth a groundbreaking coupling paradigm that combines model-driven low-rank representation (LRR) methods with data-driven deep learning techniques by learning disentangled priors (LDP). LDP seeks to capture complete priors for effectively modeling the background, thereby extracting anomalies from hyperspectral images more accurately. LDP follows a model-driven deep unfolding architecture, where the prior knowledge is separated into the explicit low-rank prior formulated by expert knowledge and implicit learnable priors by means of deep networks. The internal relationships between explicit and implicit priors within LDP are elegantly modeled through a skip residual connection. Furthermore, we provide a mathematical proof of the convergence of our proposed model. Our experiments, conducted on multiple widely recognized datasets, demonstrate that LDP surpasses most of the current advanced HAD techniques, exceling in both detection performance and generalization capability.

Multispectral image (MS) and panchromatic image (PAN) fusion, which is also named as multispectral pansharpening, aims to obtain MS with high spatial resolution and high spectral resolution. However, due to the usual neglect of noise and blur generated in the imaging and transmission phases of data during training, many deep learning (DL) pansharpening methods fail to perform on the dataset containing noise and blur. To tackle this problem, a variational optimization-guided two-stage network (VOGTNet) for multispectral pansharpening is proposed in this work, and the performance of variational optimization (VO)-based pansharpening methods relies on prior information and estimates of spatial-spectral degradation from the target image to other two original images. Concretely, we propose a dual-branch fusion network (DBFN) based on supervised learning and train it by using the datasets containing noise and blur to generate the prior fusion result as the prior information that can remove noise and blur in the initial stage. Subsequently, we exploit the estimated spectral response function (SRF) and point spread function (PSF) to simulate the process of spatial-spectral degradation, respectively, thereby making the prior fusion result and the adaptive recovery model (ARM) jointly perform unsupervised learning on the original dataset to restore more image details and results in the generation of the high-resolution MSs in the second stage. Experimental results indicate that the proposed VOGTNet improves pansharpening performance and shows strong robustness against noise and blur. Furthermore, the proposed VOGTNet can be extended to be a general pansharpening framework, which can improve the ability to resist noise and blur of other supervised learning-based pansharpening methods. The source code is available at https://github.com/HZC-1998/VOGTNet.

In the expansive domain of computer vision, a myr-iad of pretrained models are at our disposal. However, most of these models are designed for natural RGB images and prove inadequate for spectral remote sensing (RS) images. Spectral RS images have two main traits: (1) multiple bands capturing diverse feature information, (2) spatial alignment and consistent spectral sequencing within the spatial-spectral dimension. In this paper, we introduce Spatial-SpectralMAE (S2MAE), a specialized pretrained architecture for spectral RS imagery. S2MAE employs a 3D transformer for masked autoencoder modeling, inte-grating learnable spectral-spatial embeddings with a 90% masking ratio. The model efficiently captures local spec-tral consistency and spatial invariance using compact cube tokens, demonstrating versatility to diverse input characteristics. This adaptability facilitates progressive pretraining on extensive spectral datasets. The effectiveness of S2MAE is validated through continuous pretraining on two sizable datasets, totaling over a million training images. The pretrained model is subsequently applied to three dis-tinct downstream tasks, with in-depth ablation studies conducted to emphasize its efficacy.

SAR-to-optical remote sensing translator neural networks are mostly trained on flat areas, avoiding SAR geometrical distortion issues in steeply sloped areas. Their degraded performance under such topology severely limits the ability to detect disasters such as landslides in cloud covered areas. In this paper, we first propose a new SAR-DEM-optical dataset in mountainous regions to improve the performance of SAR-to-optical image translators under these extreme conditions. Then we upgrade SARDINet (SAR Distorted Image translator Network) model previously developed for urban areas, to take a Digital Elevation Model (DEM) together with the SAR image as input and perform translation in a natural mountain environment. Several fusion strategies are explored to efficiently merge SAR and DEM images: late fusion, early fusion and an intermediate fusion based on balanced separable convolutions. These approaches show improvements in distorted regions compared to the original SARDINet and two standard adversarial networks - Pix2pix and CycleGAN.

In this paper, a representative set of state-of-the-art methods for hyperspectral pansharpening, comprising both model- and deep learning-based ones, are reviewed and compared on four datasets from the PRISMA mission. The experimental analysis has been carried out using the most credited pansharpening quality indexes, complemented by a subjective visual inspection of sample results. The obtained outcomes have provided us a preview of the strengths and weaknesses of the latest solutions to the problem at hand, paving the way for future research lines from both the methodological and quality assessment perspectives.

Despite the wide application of deep learning in hyperspectral classification, variations in data collection conditions can lead to domain shift between the training and testing datasets. Traditional hyperspectral classification methods are adversely affected by these distribution differences, resulting in poor generalization performance on the testing set. To overcome this challenge, we present an optimized unsupervised domain adaptation approach based on causal invariance. Our method assumes a causal relationship to reflect the effects of changes in class information and domain information on samples. Based on this causal relationship, we construct a network to separate class-related and domain-related features. To further reduce the negative transfer caused by distribution differences, our model introduces intra-class feature consistency. As a result, our method improves the performance of the model on the target domain. Experimental results on two public hyperspectral datasets demonstrate the superior effectiveness of our method.

Due to the high data demand of machine learning algorithms, multiple datasets are emerging in remote sensing. But these datasets are costly and time consuming to annotate especially for change detection or natural phenomena monitoring. In particular, early warning systems on slow-moving disasters are lacking of training datasets as they require both geomorphological and SAR interferometry expertise. In this paper, (i) we propose a novel InSAR dataset for Slow SLIding area DEtection (ISSLIDE) with machine learning algorithms. The latter consists of manually annotated patches of generated interferograms over slow moving areas. (ii) We implement the segmentation of ISSLIDE interferograms with classical deep learning approaches. FCN, DeepLabV3 and U-Net-like architectures are explored to serve as baseline for future works. To the best of our knowledge, this is the first dataset adapted to machine learning and targeting slow sliding area detection.

Extracting rock objects from the surface of celestial bodies in deep space exploration environments is crucial for self-service path planning, navigation of detectors, and regional information evaluation. Most existing image saemantic segmentation frameworks decrease the spatial resolution of the feature maps as networks deepen, resulting in limitations in detecting small targets and the inability to accurately segment boundary regions. In this letter, we propose a novel semantic segmentation network based on U-Net network and Transformer for deep space rock images, referred to as Diamond-Unet. This model integrates overcomplete and undercomplete branches and incorporates a global-local feature extraction (GLFE) module based on Transformer and CNN technologies to effectively capture discriminative information. Furthermore, an innovative feature cross-fusion path (FCFP) is introduced to enhance information exchange between the dual-branch networks, enabling the capture of both fine-grained details and coarse-grained semantics in the full-scale image segmentation architecture. Experimental results demonstrate that the Diamond-Unet achievesMIoUscores of 79.32% and 93.43% on two public datasets, which are superior to the compared methods.

Video coding for machines (VCM) is a rapidly growing field dedicated to bridging the gap between video and feature coding. For storage-intensive aerial videos, VCM offers valuable insights into a more efficient coding paradigm. However, the frequent occurrence of small objects poses a challenge to VCM, with limited distinctive features and inherent distortion in the reconstructed videos. To address this issue, we propose small object-aware VCM (SOAVCM), a joint video and feature coding approach that handles small objects. Particularly, the video coding incorporates a feature-guided residual (FGR) codec to preserve the small objects, utilizing features obtained from feature coding. Simultaneously, feature coding employs the motion vector (MV) estimated in video coding to generate compact high-level features. By leveraging the inherent synergy between features and MVs, SOAVCM significantly enhances overall coding efficiency. Experimental results demonstrate that SOAVCM outperforms several deep-learning-based methods and traditional coding standards in video coding. Moreover, the encoded feature representation improves detection accuracy and achieves substantial bitrate savings.

Model-based target decomposition method has been widely applied due to its clear physical scattering significance. However, after establishing decomposition basis, the process of solving the scattering components and parameters is usually underdetermined, which will lead to the issues such as component negative power and overestimation. For this problem, this letter examines the target decomposition task from the perspective of deep learning and proposes an adversarial decomposition feature learning (ADFL) model. This model could learn decomposition features suitable for current terrain characteristics according to input data. At the same time, the model-based adversarial feature prior is embedded in ADFL to maintain the physical scattering meanings. On real PolSAR datasets, the learned features of proposed model are well correlated with real terrain scattering characteristics. Further, it avoids negative decomposition features and make more accurate fitting of scattering components, effectively alleviating the above problems.

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.

Recently, cross-scene hyperspectral image classification (HSIC) has attracted increasing attention, alleviating the dilemma of no labeled samples in the target domain. Although collaborative source and target training has dominated this field, training effective feature extractors and overcoming intractable domain gaps remains challenging. To cope with this issue, we propose a multi-level unsupervised domain adaptation (MLUDA) framework, which comprises image-, feature-, and logic-level alignment between domains to fully investigate the comprehensive spectral-spatial information. Specifically, at the image level, we propose an innovative domain adaptation method named GuidedPGC based on classic image matching techniques and the guided filter. The adaptation results are physically explainable with intuitive visual observations. Regarding the feature level, we design a multi-branch cross attention structure (MBCA) specifically for HSIC, which enhances the interaction between the features from the source and target domains through dot-product attention. Finally, at the logic level, we adopt a supervised contrastive learning (SCL) approach that incorporates a pseudo-label strategy and local maximum mean discrepancy loss, increasing inter-class distance across diverse domains and further improving the classification performance. Experimental results on three benchmark cross-scene datasets demonstrate that our proposed method consistently outperforms the compared approaches. The source code is available at https://github.com/cfcys/MLUDA.

Sentinel-5P provides excellent spatial information, but its resolution is insufficient to characterise the complex distribution of air contaminants within limited areas. As physical constraints prevent significant advances beyond its nominal resolution, employing processing techniques like single-image super-resolution can notably contribute to both research and air quality monitoring applications. This study presents the very first use of such methodologies on Sentinel-5P data. We demonstrate that superior results may be obtained if the degrading filter used to simulate pairs of low- and high-resolution images is tailored to the acquisition technology at hand, an issue frequently ignored in the scientific literature on the subject. Because of this, as well as the fact that these data have never been deployed in any previous studies, most of the work theoretical contribution is the estimation of the degradation model of TROPOMI, the sensor mounted on Sentinel-5P. Leveraging this model—which is essential for applications involving super-resolution—we additionally improve a well-known deconvolution-based strategy and present a brand-new neural network that outperforms both traditional super-resolution techniques and well-established neural networks in the field. The findings of this study, that are supported by experimental tests on real Sentinel-5P radiance images, using both full-scale and reduced-scale protocols, offer a baseline for enhancing algorithms that are driven by the understanding of the imaging model and provide an efficient way of evaluating innovative approaches on all the available images. The code is available at https://github.com/alcarbone/S5P_SISR_Toolbox.

In recent years, multimodal remote sensing data classification (MMRSC) has evoked growing attention due to its more comprehensive and accurate delineation of Earth’s surface compared to its single-modal counterpart. However, it remains challenging to capture and integrate local and global features from single-modal data. Moreover, how to fully excavate and exploit the interactions between different modalities is still an intricate issue. To this end, we propose a novel dual-branch transformer-based framework named deep symmetric fusion transformer (DSymFuser). Within the framework, each branch contains a stack of local-global mixture (LGM) blocks, to extract hierarchical and discriminative single-modal features. In each LGM block, a local-global feature mixer with learnable weights is specifically devised to adaptively aggregate the local and global features extracted with a convolutional neural network (CNN)–transformer network. Furthermore, we innovatively design a symmetric fusion transformer (SFT) that trails behind each LGM block. The elaborately designed SFT symmetrically facilitates cross-modal correlation excavation, comprehensively exploiting the complementary cues underlying heterogeneous modalities. The hierarchical construction of the LGM and SFT blocks enables feature extraction and fusion in a multilevel manner, further promoting the completeness and descriptiveness of the learned features. We conducted extensive ablation studies and comparative experiments on three benchmark datasets, and the experimental results validated the effectiveness and superiority of the proposed method. The source code of the proposed method will be available publicly athttps://github.com/HaixiaBi1982/DSymFuser.

Many existing multiobject tracking (MOT) methods tend to model each object’s feature individually. However, under acute viewpoint variation and occlusion, there may exist significant differences between the current and historical features of objects, which easily leads to object loss. To alleviate these issues, the topological relationships (i.e., geometric shapes formed by objects) should be modeled as a supplement to individual object features to maintain stability. In this article, we propose a novel MOT framework, which consists of a frame graph and association graph, to leverage the topological relationships both spatially and temporally. Technically, the frame graph models distance and angle among objects to resist viewpoint change, while the association graph utilizes the interframe temporal consistency of topological features to recover occluded objects. Extensive experiments on mainstream datasets demonstrate the effectiveness.

Existing cross-domain few-shot learning (FSL) methods for hyperspectral image (HSI) classification have garnered widespread attention due to their excellent performance in recognizing novel classes. To mitigate domain shift, researchers focus on designing sophisticated domain adaptation (DA) modules to directly apply biased metaknowledge in the target domain (TD). However, this paradigm proves somewhat inadequate in the face of significant differences in distribution. To cope with this dilemma, we adopted a new mindset of treating metaknowledge extraction and debiasing from the source domain (SD) as a synergistic process and proposed a cross-domain FSL framework based on decoupled knowledge distillation for HSI classification (HSIC). In general, to efficiently acquire and utilize unbiased metaknowledge, this framework centralizes on a knowledge distillation (KD) strategy. Through the effective information transfer process, the extraction and debiasing of metaknowledge were integrated into a comprehensive and productive process. Simultaneously, to release the constraints imposed by the coupled logits in the KD process on the knowledge interaction, the decoupled logit interaction (DLI) module is employed in the framework. This module decouples the traditional KD into two controllable components, making a more balanced and comprehensive interaction of task-related knowledge and data-intrinsic knowledge between models. Moreover, to facilitate the extraction of critical discriminative metaknowledge from the abundant redundant information in HSI, the discriminative information refinement (DIR) module is designed to develop distinctive features for similar bands. Extensive experiments on three public HSI datasets exhibited the superior performance of the proposed cross-domain few-shot learning method based on decoupled knowledge distillation for HSIC (DKD-FSL) method in comparison with seven state-of-the-art approaches.

Deep learning (DL) has been widely applied to hyperspectral image (HSI) classification, owing to its promising feature learning and representation capabilities. However, limited by the spatial resolution of sensors, existing DL-based classification approaches mainly focus on pixel-level spectral and spatial information extraction through complex network architecture design while ignoring the existence of mixed pixels in actual scenarios. To tackle this difficulty, we propose a novel dual-branch subpixel-guided network for HSI classification, called DSNet, which automatically integrates subpixel information and convolutional class features by introducing a deep autoencoder unmixing architecture to enhance classification performance. DSNet is capable of fully considering physically nonlinear properties within subpixels and adaptively generating diagnostic abundances in an unsupervised manner to achieve more reliable decision boundaries for class label distributions. The subpixel fusion module is designed to ensure high-quality information fusion across pixel and subpixel features, further promoting stable joint classification. Experimental results on three benchmark datasets demonstrate the effectiveness and superiority of DSNet compared with state-of-the-art DL-based HSI classification approaches. The codes will be available athttps://github.com/hanzhu97702/DSNet, contributing to the remote sensing community.

Cross-scene image classification aims to transfer prior knowledge of ground materials to annotate regions with different distributions and reduce hand-crafted cost in the field of remote sensing. However, existing approaches focus on single-source domain generalization to unseen target domains, and are easily confused by large real-world domain shifts due to the limited training information and insufficient diversity modeling capacity. To address this gap, we propose a novel multi-source collaborative domain generalization framework (MS-CDG) based on homogeneity and heterogeneity characteristics of multi-source remote sensing data, which considers data-aware adversarial augmentation and model-aware multi-level diversification simultaneously to enhance cross-scene generalization performance. The data-aware adversarial augmentation adopts an adversary neural network with semantic guide to generate MS samples by adaptively learning realistic channel and distribution changes across domains. In views of cross-domain and intra-domain modeling, the model-aware diversification transforms the shared spatial-channel features of MS data into the class-wise prototype and kernel mixture module, to address domain discrepancies and cluster different classes effectively. Finally, the joint classification of original and augmented MS samples is employed by introducing a distribution consistency alignment to increase model diversity and ensure better domain-invariant representation learning. Extensive experiments on three public MS remote sensing datasets demonstrate the superior performance of the proposed method when benchmarked with the state-of-the-art methods.

Standard hyperspectral (HS) pansharpening utilizes panchromatic (PAN) images to improve the connected low-resolution HS (LRHS) images to the spatial resolutions of PANs; while arbitrary-resolution hyperspectral (ARHS) pansharpening aims to use PANs to enhance LRHS images to any desired spatial resolutions. For the challenging task of ARHS pansharpening, one of major obstacles is how to generalize the single pansharpening model learned under predetermined training scales to any pansharpening scales for future data. As implicit neural representations (INRs) have a potential to approximate continuous functions, they offer a possible alternative way to naturally resolve ARHS pansharpening. In this paper, we develop two spectral-spatial INRs for ARHS pansharpening: one is a naive pansharpening INR (NaivePINR); the other is a dynamic pansharpening INR (DynamicPINR). The former builds a novel spectral-spatial encoding to produce spectral-spatial priors of observed scenes and uses a spectral-spatial query mapping to reconstruct fine spectral details and spatial details. The latter establishes a innovative two-fold tuning mechanism to dynamically adjust both the spectral-spatial encoding and the spectral-spatial query mapping. Experimental results on serval datasets verify the excellent performances of the proposed pansharpening INRs.

Land use/land cover classification with multimodal data has attracted increasing attention. For hyperspectral images (HSIs) and light detection and ranging (LiDAR) data, the combination of them can make the classification more accurate and robust. However, how to effectively utilize their respective strengths and integrate them with the classification task is still a challenging problem. In this article, a multilevel attention dynamic-scale network (MADNet) is proposed. First, in the feature extraction stage, the two modalities are divided into two branches with different scales, which are then fed into the convolutional neural networks (CNNs) to learn shallow features. Then, considering the characteristics of the HSI, a spectral angle attention module (SAAM) with low-level attention is designed to highlight surrounding pixels that have similar spectra to the central pixel of the patch. After that, a dynamic-scale selection module (DSSM) is proposed to screen an appropriate scale for the patches by pixel similarity analysis. Next, combining the Transformer and the CNN, a global-local cross-attention module (GLCAM) is devised to investigate the fused deep-level multimodal features. Distinct from the vanilla Transformer, the GLCAM deploys a distance-weight operator to decrease the redundancies at long distances and effectively reduce misclassifications. Extensive experiments on three paired HSI and LiDAR datasets demonstrate that the proposed MADNet has certain advantages over the existing methods.

By fusing a low-resolution hyperspectral image (LrMSI) with an auxiliary high-resolution multispectral image (HrMSI), hyperspectral image super-resolution (HISR) can generate a high-resolution hyperspectral image (HrHSI) economically. Despite the promising performance achieved by deep learning (DL), there are still two challenges remaining to be solved. First, most DL-based methods heavily rely on large-scale training triplets, which reduces them to limited generalization and poor practicability in real-world scenarios. Second, existing methods pursue higher performance by designing complex structures from off-the-shelf components while ignoring inherent information from the degradation model, hence leading to insufficient integration of domain knowledge and lower interpretability. To address those drawbacks, we propose a model-informed multi-stage unsupervised network, M2U-Net for short, by leveraging both deep image prior (DIP) and degradation model information. Generally, M2U-Net is built with a three-stage scheme, i.e., degradation information learning (DIL), initialized image establishment (IIE), and deep image generation (DIG) stages. The first stage is to exploit the deep information of the degradation model via a tiny network whose parameters and outputs will serve as guidance for the following two stages. Instead of feeding uninformed noise as input for stage three, IIE stage aims to establish an initialized input with expressive HrHSI-relevant information by resorting to a spectral mapping learning network, thus facilitating the extraction of prior information and further magnifying the potential of DIP for high-quality reconstruction. Last, we propose a dual U-shape network as a powerful regularizer to capture image statistics, in which two U-Nets are coupled together by cross-attention guidance (CAG) module to separately achieve spatial feature extraction and final image generation. The CAG module can incorporate abundant spatial information into the reconstruction process and hence guide the network toward a more plausible generation. Extensive experiments demonstrate the effectiveness of our proposed M2U-Net in terms of quantitative evaluation and visual quality. The code will be available at https://github.com/JiaxinLiCAS.

Current hyperspectral anomaly detection (HAD) benchmark datasets suffer from low resolution, simple background, and small size of the anomalies. These factors also limit the performance of the well-known low-rank representation (LRR) models in terms of robustness on the separation of background and target features and the reliance on manual parameter selection. To this end, we build a new HAD benchmark dataset for improving the robustness in complex scenarios, AIR-HAD for short, and propose an interpretable network with deep unfolding a binary subspace learning, named LRR-Net+, which is capable of spectrally decoupling the background structure and object properties in a more generalized fashion and eliminating the bias introduced by vital interference targets simultaneously. In addition, LRR-Net+ integrates the solution process of the alternating direction method of multipliers (ADMM) optimizer with the deep network, guiding its search process and imparting a level of interpretability to parameter optimization. Additionally, the integration of physical models with DL techniques eliminates the need for manual parameter tuning. The manually tuned parameters are seamlessly transformed into trainable parameters for deep neural networks, facilitating a more efficient and automated optimization process. Extensive experiments conducted on the AIR-HAD dataset show the superiority of our LRR-Net+ in terms of detection performance and generalization ability, compared to top-performing competitors. Furthermore, our AIR-HAD benchmark datasets will be made available freely and openly athttps://github.com/danfenghong/IEEE_TGRS_LRR-Net.

Change detection (CD) is a critical remote sensing technique for identifying changes in the Earth’s surface over time. The outstanding substance identifiability of hyperspectral images (HSIs) has significantly enhanced the detection accuracy, making hyperspectral CD (HCD) an essential technology. The detection accuracy can be further upgraded by leveraging the graph structure of HSIs, motivating us to adopt the graph neural networks (GNNs) in solving HCD. For the first time, this work introduces a quantum deep network (QUEEN) into HCD. Unlike GNN and CNN, both extracting the affine-computing features, QUEEN provides fundamentally different unitary-computing features. We demonstrate that through the unitary feature extraction procedure, QUEEN provides radically new information for deciding whether there is a change or not. Hierarchically, a graph feature learning (GFL) module exploits the graph structure of the bitemporal HSIs at the superpixel level, while a quantum feature learning (QFL) module learns the quantum features at the pixel level, as a complementary to GFL by preserving pixel-level detailed spatial information not retained in the superpixels. In the final classification stage, a quantum classifier is designed to cooperate with a traditional fully connected classifier. The superior HCD performance of the proposed QUEEN-empowered GNN (i.e., QUEEN-$\mathcal {G}$) will be experimentally demonstrated on real hyperspectral datasets.

Deep feature learning methods have shown significant advantages over handcrafted feature-based methods in remote sensing image matching and registration. Existing deep learning methods usually introduce complex modules into the deep convolutional network for more robust feature learning. However, they usually require high computation and memory resources for the computing device and have expensive time costs for image registration. As a basic image-processing task, it is crucial to build a lightweight matching network (LM-Net) for fast and accurate image matching and registration. Unfortunately, the image-matching performance will decrease significantly when we directly compress the deep model to a lightweight one. This article proposes an LM-Net based on the knowledge distillation (KD) learning framework for remote sensing image matching and registration. We first build an LM-Net with three convolutional layers. Then, this article proposes an effective KD approach for network optimization, which transfers the effective knowledge from the deep matching network to LM-Net to improve image-matching performances. Specifically, this article considers the useful information in the instance samples and the relation information between samples. It designs the feature and feature relation distillation learning for LM-Net training. Extensive experimental results and analysis have shown the effectiveness and advantages of the proposed LM-Net. LM-Net can reduce the number of parameters and computational complexity of the matching network. Meanwhile, LM-Net can significantly decrease the time cost and achieve results comparable to those of the deep model. It reduces the average image registration time by 42% on remote sensing image matching and registration. Additionally, LM-Net generalizes well on other multimodal remote sensing images.

Multimodal remote sensing image registration is crucial for multimodal information fusion and applications. The significant nonlinear appearance difference between multimodal images caused by the various imaging mechanisms dramatically increases the challenge of image registration. This article proposes an adaptive frequency feature filtering network (F3Net) for cross-modal remote sensing image registration. On the one hand, F3Net explicitly explores the useful frequency components across modal images based on multilevel deep features. On the other hand, F3Net can take advantage of the nonlocal receptive fields by frequency modulation for feature learning and boosting image registration performances. F3Net inserts frequency feature filtering (F3) modules in multilevel deep features. Specifically, F3Net first performs the fast Fourier transform (FFT) for deep features. Then, F3Net designs a frequency attention (FA) module to adaptive enhance the shared and discriminative frequency features between multimodal images while suppressing the frequency components that hinder the cross-modal image registration. In addition, F3Net adopts multiscale frequency filtering fusion to facilitate discriminative feature learning, including global frequency feature filtering (GF3) based on the global image spectrum and local frequency feature filtering (LF3) based on the spectrum of stacked image regions. Experimental results on many remote sensing images have demonstrated the efficiency of the F3Net on multimodal image registration.

Spectral pixels are often a mixture of the pure spectra of the materials, called endmembers, due to the low spatial resolution of hyperspectral sensors, double scattering, and intimate mixtures of materials in the scenes. Unmixing estimates the fractional abundances of the endmembers within the pixel. Depending on the prior knowledge of endmembers, linear unmixing can be divided into three main groups: supervised, semi-supervised, and unsupervised (blind) linear unmixing. Advances in image processing and machine learning substantially affected unmixing. This paper provides an overview of advanced and conventional unmixing approaches. Additionally, we draw a critical comparison between advanced and conventional techniques from the three categories. We compare the performance of the unmixing techniques on three simulated and one real dataset. The experimental results reveal the advantages of different unmixing categories for different unmixing scenarios. Moreover, we provide an open-source Python-based package available at https://github.com/BehnoodRasti/HySUPP to reproduce the results.

In this article, we introduce a novel linear model tailored for semisupervised/library-based unmixing. Our model incorporates considerations for library mismatch while enabling the enforcement of the abundance sum-to-one constraint (ASC). Unlike conventional sparse unmixing methods, this model involves nonconvex optimization, presenting significant computational challenges. We demonstrate the efficacy of alternating direction method of multipliers (ADMM) in cyclically solving these intricate problems. We propose two semisupervised unmixing approaches, each relying on distinct priors applied to the new model in addition to the ASC: sparsity prior and convexity constraint. Our experimental results validate that enforcing the convexity constraint outperforms the sparsity prior for the endmember library. These results are corroborated across three simulated datasets (accounting for spectral variability and varying pixel purity levels) and the Cuprite dataset. In addition, our comparison with conventional sparse unmixing methods showcases considerable advantages of our proposed model, which entails nonconvex optimization. Notably, our implementations of the proposed algorithms—fast semisupervised unmixing (FaSUn) and sparse unmixing using soft shrinkage (SUnS)—prove considerably more efficient than traditional sparse unmixing methods. SUnS and FaSUn were implemented using PyTorch and provided in a dedicated Python package called FaSUn, which is open-source and available athttps://github.com/BehnoodRasti/FUnmix.

Hyperspectral anomaly detection aims at distinguishing targets of interest from the background without prior knowledge. Although low-rank representation (LRR)-based methods have been broadly applied in anomaly detection tasks, how to approximate the penalties in LRR-based methods more precisely is still a problem that needs to be further investigated. To this end, this article designs a unified nonconvex framework called hyperspectral anomaly detection via generalized shrinkage mappings (HADGSMs) to better approximate the LRR-based methods. The core of the proposed framework is to design new nonconvex penalties to approximate the group sparsity,$l_{0}$gradient, and low-rankness penalties in the LRR-based anomaly detection models, which can be efficiently minimized by means of generalized shrinkage mappings (GSMs). Then, an efficient alternating direction method of multipliers (ADMM) is developed to handle the proposed model. Experiments conducted on several real hyperspectral datasets demonstrate the superiority and effectiveness of the proposed framework in enhancing detection performance with respect to state-of-the-art methods.

Gathering extensive labeled data during Mars missions is costly and unrealistic, especially considering the complex and unpredictable Martian environment where new and unfamiliar scenes may emerge. Traditional Mars scene classification (MSC) methods depend heavily on large amounts of labeled data, which makes it impractical to recognize previously unseen scene classes without the necessary labeled examples. In addition, the significant computational demands and parameter requirements of modern models also pose challenges for their integration into resource-constrained systems used in Mars exploration. To address these issues, we propose a zero-shot MSC (ZSMSC) framework, which is able to categorize unseen Martian image scenes without the prior acquisition of vast visual examples. Specifically, the framework combines lightweight model design with knowledge distillation (KD) techniques, known as KDMSC, to streamline complex zero-shot learning (ZSL) models. It employs a KD loss that captures essential knowledge through the training of the teacher model from scratch, thereby improving the zero-shot classification performance of the student model. Consequently, the lightweight student model is tailored for deployment on devices with limited resources while fulfilling the requirements of the ZSMSC tasks. Moreover, to support the ZSMSC initiative, we developed a dataset named ZSMars to further advance this field. Experimental results indicate that our model excels in the ZSMSC tasks while maintaining low computational complexity and storage requirements.

Unsupervised domain adaptation (UDA) reduces domain shifts between distributions to enable model generalization to new scenarios. Adversarial domain adaptation (DA) is an effective approach that extracts domain-invariant features through adversarial learning, but such methods often neglect the influence of category differences on domain discrimination. To solve this problem, we construct a new unsupervised domain adaption hyperspectral image (HSI) classification method. The proposed method consists of two modules, namely, the pairing domain discrimination learning module and the multilevel mutual information maximization module. We propose to construct the sample pair as the input of the domain discriminator. We introduce a new label to the sample pair according to the labels of the two samples and use the relationship between samples to reduce the impact of sample category differences on domain discrimination. When extracting the shared features of the two domains, it will inevitably cause the loss of task-related information. This information is retained by maximizing the proposed multilevel mutual information. The experimental results on different datasets show the effectiveness of our method.

In hyperspectral cross-scene classification tasks, it is often challenging to obtain target domain samples during the training phase. Therefore, models need to be trained on one or multiple source domains and achieve good generalization performance on unknown target domains, known as domain generalization. The presence of domain shift limits the model’s generalization across different domains, while the unknown target domain makes it difficult to accurately characterize the distribution differences between domains. To address this issue, we propose a generalization network based on nonlinear sample generation. The network divides the sample features into invariant features and variant features and generates samples by applying nonlinear transformations to the variant features. To ensure the quality of the generated samples, we introduce contrastive learning into the model. It ensures consistency in similarity between the generated samples and the source samples while maintaining a certain degree of dissimilarity. Experiments conducted on four cross-domain adaptive scenarios demonstrate the superior performance of our proposed method.

In large-scale disaster events, the planning of optimal rescue routes depends on the object detection ability at the disaster scene, with one of the main challenges being the presence of dense and occluded objects. Existing methods, which are typically based on the RGB modality, struggle to distinguish targets with similar colors and textures in crowded environments and are unable to identify obscured objects. To this end, we first construct two multimodal dense and occlusion vehicle detection datasets for large-scale events, utilizing RGB and height map modalities. Based on these datasets, we propose a multimodal collaboration network for dense and occluded vehicle detection, MuDet for short. MuDet hierarchically enhances the completeness of discriminable information within and across modalities and differentiates between simple and complex samples. MuDet includes three main modules: Unimodal Feature Hierarchical Enhancement (Uni-Enh), Multimodal Cross Learning (Mul-Lea), and Hard-easy Discriminative (He-Dis) Pattern. Uni-Enh and Mul-Lea enhance the features within each modality and facilitate the cross-integration of features from two heterogeneous modalities. He-Dis effectively separates densely occluded vehicle targets with significant intra-class differences and minimal inter-class differences by defining and thresholding confidence values, thereby suppressing the complex background. Experimental results on two re-labeled multimodal benchmark datasets, the 4K-SAI-LCS dataset, and the ISPRS Potsdam dataset, demonstrate the robustness and generalization of the MuDet.

Few-shot learning (FSL) has rapidly advanced in the hyperspectral image classification (HSIC), potentially reducing the need for laborious and expensive labeled data collection. Due to the limited receptive field, the convolutional neural network (CNN) struggles to capture long-range dependencies for extracting global features. Additionally, the transformer focuses on global correlation while overlooking the effective representation of local spatial and spectral features. Moreover, contrastive learning (CL) has emerged as a powerful technique for improving consistency across different augmented views of samples of the same category. To this end, we devise a novel CNN-Transformer for few-shot HSIC assisted by semisupervised contrastive learning, named CTF-SSCL, to boost the classification performance. Specifically, the cascaded CNN-Transformer incorporates a lightweight spatial-spectral interactive convolution module (LSSICM) and a multiscale transformer (MSFormer) to exploit local features from submaps and global information from the entire patch. Subsequently, the semisupervised contrastive loss, comprising unsupervised and supervised components, serves as an auxiliary to optimize the model with the classification loss. Wherein, recognizing the unified spectral-spatial information in HSI, we propose a spectral feature shift strategy (SFSS) to create sample pairs for the unsupervised CL, utilizing unsupervised contrastive loss among groups of samples with identical labels. Extensive experiments on four standard benchmarks demonstrate the effectiveness of the proposed CTF-SSCL with varying amounts of labeled samples. The code will be available online athttps://github.com/B-Xi/CTF-SSCL.

The spectral super-resolution (SpeSR) from multispectral images (MSIs) to hyperspectral images (HSIs) can bring rich spectral information. The deep learning-based methods have demonstrated their powerful ability for the SpeSR task, which requires the paired HSI/MSI to train the model. However, HSIs and MSIs are always obtained at different times and under different imaging conditions, covering different areas. To address this issue, in this paper, a framework named BliEstGAN based on the generative adversarial network (GAN) is proposed to estimate the spectral resolution degradation between unpaired HSIs and MSIs that can be used for the blind SpeSR. Specifically, each MSI imaging sensor has its own unique spectral sampling process, which can be modeled as a spectral degradation from its paired HSI. Different spectral degradations can be discriminated by the deep model. Therefore, the generator of the GAN is used to estimate the spectral degradation from HSIs to MSIs, and the discriminator of the GAN is adopted to distinguish whether the estimated and real spectral degradation are similar. The large difference in spatial resolution between MSIs and HSIs makes them easy to discriminate against. Therefore, smooth hyperspectral and multispectral patches are extracted from HSIs and MSIs to eliminate this difference in spatial resolution. Furthermore, according to the imaging sensor mechanism, some special regularization terms are designed for the generator to guarantee its correct convergence. Finally, the estimated spectral resolution degradation can be adopted to generate HSI/MSI pairs for the supervised learning-based SpeSR methods. Experimental results demonstrate the effectiveness of the proposed method.

The most cost-effective way to obtain a high spatial resolution hyperspectral image (HrHSI) is to fuse a low spatial resolution hyperspectral image (LrHSI) and corresponding high spatial resolution multispectral image (HrMSI). This article proposes a generalizable unsupervised deep fusion method based on spectral-spatial collaborative constraint to address LrHSI and HrMSI fusion task. First, in view of the limitations of the current spectral-spatial downsampled model, the group convolution enhancement (GCE) module is designed to eliminate the radiometric difference between the images to be fused. Second, to enhance the model’s feature extraction ability, this article introduces the design of the spatial, channel, and filter 3-D attention factor dynamic convolutional kernel (SCFConv). In order to verify the proposed method, we compared and evaluated our method with traditional methods and unsupervised deep learning methods using both simulated and real onboard data, respectively. In the absence of HrHSI validation images in real scenarios, we evaluate the performance of different fusion models through classification results. The experimental results demonstrate the effectiveness of the proposed model and the practical value of the fusion results (the onboard data produced by ours are available athttps://drive.google.com/drive/folders/1JLCCB6ld5R49HDLN5SsMISx1d0fuqRjO).

Segment anything model (SAM) has been widely applied to various downstream tasks for its excellent performance and generalization capability. However, SAM exhibits three limitations related to remote sensing semantic segmentation task: 1) the image encoders excessively lose high-frequency information, such as object boundaries and textures, resulting in rough segmentation masks; 2) due to being trained on natural images, SAM faces difficulty in accurately recognizing objects with large-scale variations and uneven distribution in remote sensing images; 3) the output tokens used for mask prediction are trained on natural images and not applicable to remote sensing image segmentation. In this paper, we explore an efficient paradigm for applying SAM to the semantic segmentation of remote sensing images. Furthermore, we propose MeSAM, a new SAM fine-tuning method more suitable for remote sensing images to adapt it to semantic segmentation tasks. Our method first introduces an inception mixer into the image encoder to effectively preserve high-frequency features. Secondly, by designing a mask decoder with remote-sensing correction and incorporating multiscale connections, we make up the difference in SAM from natural images to remote sensing images. Experimental results demonstrated that our method significantly improves the segmentation accuracy of SAM for remote sensing images, outperforming some state-of-the-art methods. The code will be available at https://github.com/Magic-lem/MeSAM.

Image registration and change detection are crucial for multitemporal remote sensing image analysis. The images should be registered before the change information detection. Existing deep learning methods have shown significant advantages in image registration and change detection tasks. They usually design two independent task-specific deep networks for image registration and change detection, respectively. These independent deep networks will learn from scratch and rely on many task-specific labeled training datasets. This article finds that image registration and change detection have similar learning mechanisms, which focus on extracting discriminative features. Inspired by this, we propose a Unified image Registration and Change detection Network (URCNet) that can perform image alignment and change information detection through a single network. Additionally, this article proposes various deep collaborative learning methods for URCNet optimization, which enforce that the URCNet can effectively support remote sensing image registration and change detection simultaneously. Extensive experiments demonstrate the effectiveness of the proposed URCNet for image registration and change detection, which can achieve comparable and better results with task-specific and more complex deep networks. The proposed URCNet can support multitasks based on the same scene images, different scene images, and even multimodal images. Moreover, URCNet shows significant advantages over other deep networks in change detection under limited labeled datasets.

Multimodal data provide complementary information of a natural phenomenon by integrating data from various domains with very different statistical properties. Capturing the intramodality and cross-modality information of multimodal data is the essential capability of multimodal learning methods. The geometry-aware data analysis approaches provide these capabilities by implicitly representing data in various modalities based on their geometric underlying structures. Also, in many applications, data are explicitly defined on an intrinsic geometric structure. Generalizing deep learning methods to the non-Euclidean domains is an emerging research field, which has recently been investigated in many studies. Most of those popular methods are developed for unimodal data. In this article, a multimodal graph wavelet convolutional network (M-GWCN) is proposed as an end-to-end network. M-GWCN simultaneously finds intramodality representation by applying the multiscale graph wavelet transform to provide helpful localization properties in the graph domain of each modality and cross-modality representation by learning permutations that encode correlations among various modalities. M-GWCN is not limited to either the homogeneous modalities with the same number of data or any prior knowledge indicating correspondences between modalities. Several semisupervised node classification experiments have been conducted on three popular unimodal explicit graph-based datasets and five multimodal implicit ones. The experimental results indicate the superiority and effectiveness of the proposed methods compared with both spectral graph domain convolutional neural networks and state-of-the-art multimodal methods.

Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.

This article addresses the problem of the building an out-of-the-box deep detector, motivated by the need to perform anomaly detection across multiple hyperspectral images (HSIs) without repeated training. To solve this challenging task, we propose a unified detector [anomaly detection network (AUD-Net)] inspired by few-shot learning. The crucial issues solved by AUD-Net include: how to improve the generalization of the model on various HSIs that contain different categories of land cover; and how to unify the different spectral sizes between HSIs. To achieve this, we first build a series of subtasks to classify the relations between the center and its surroundings in the dual window. Through relation learning, AUD-Net can be more easily generalized to unseen HSIs, as the relations of the pixel pairs are shared among different HSIs. Secondly, to handle different HSIs with various spectral sizes, we propose a pooling layer based on the vector of local aggregated descriptors, which maps the variable-sized features to the same space and acquires the fixed-sized relation embeddings. To determine whether the center of the dual window is an anomaly, we build a memory model by the transformer, which integrates the contextual relation embeddings in the dual window and estimates the relation embeddings of the center. By computing the feature difference between the estimated relation embeddings of the centers and the corresponding real ones, the centers with large differences will be detected as anomalies, as they are more difficult to be estimated by the corresponding surroundings. Extensive experiments on both the simulation dataset and 13 real HSIs demonstrate that this proposed AUD-Net has strong generalization for various HSIs and achieves significant advantages over the specific-trained detectors for each HSI.

This article focuses on end-to-end image matching through joint key-point detection and descriptor extraction. To find repeatable and high discrimination key points, we improve the deep matching network from the perspectives of network structure and network optimization. First, we propose a concurrent multiscale detector (CS-det) network, which consists of several parallel convolutional networks to extract multiscale features and multilevel discriminative information for key-point detection. Moreover, we introduce an attention module to fuse the response maps of various features adaptively. Importantly, we propose two novel rank consistent losses (RC-losses) for network optimization, significantly improving image matching performances. On the one hand, we propose a score rank consistent loss (RC-S-loss) to ensure that the key points have high repeatability. Different from the score difference loss merely focusing on the absolute score of an individual key point, our proposed RC-S-loss pays more attention to the relative score of key points in the image. On the other hand, we propose a score-discrimination RC-loss to ensure that the key point has high discrimination, which can reduce the confusion from other key points in subsequent matching and then further enhance the accuracy of image matching. Extensive experimental results demonstrate that the proposed CS-det improves the mean matching result of deep detector by 1.4%-2.1%, and the proposed RC-losses can boost the matching performances by 2.7%-3.4% than score difference loss. Our source codes are available at https://github.com/iquandou/CS-Net.

It is generally known that pan-sharpening is fundamentally a PAN-guided multispectral (MS) image super-resolution problem that involves learning the nonlinear mapping from low-resolution (LR) to high-resolution (HR) MS images. Since an infinite number of HR-MS images can be downsampled to produce the same corresponding LR-MS image, learning the mapping from LR-MS to HR-MS image is typically ill-posed and the space of the possible pan-sharpening functions can be extremely large, making it difficult to estimate the optimal mapping solution. To address the above issue, we propose a closed-loop scheme that learns the two opposite mapping including the pan-sharpening and its corresponding degradation process simultaneously to regularize the solution space in a single pipeline. More specifically, an invertible neural network (INN) is introduced to perform a bidirectional closed-loop: the forward operation for LR-MS pan-sharpening and the backward operation for learning the corresponding HR-MS image degradation process. In addition, given the vital importance of high-frequency textures for the Pan-sharpened MS images, we further strengthen the INN by designing a specified multiscale high-frequency texture extraction module. Extensive experimental results demonstrate that the proposed algorithm performs favorably against state-of-the-art methods qualitatively and quantitatively with fewer parameters. Ablation studies also verify the effectiveness of the closed-loop mechanism in pan-sharpening. The source code is made publicly available at https://github.com/manman1995/pan-sharpening-Team-zhouman/.

This paper introduces an open-source hyperspectral unmixing Python package called HySUPP. Hyperspectral unmixing can be divided into three main categories considering the prior knowledge of the endmembers; supervised, semi-supervised, and unsupervised (blind) unmixing. HySUPP includes more than 20 Python-based unmixing approaches from different categories. In the experimental section, we use a few candidates of every category and compare different unmixing approaches. In the experimental section, a challenging simulated dataset without pure pixels is used. The results confirm the advantages of deep learning-based unmixing approaches compared to conventional techniques for all categories in terms of abundance root mean square error. Additionally, blind unmixing approaches outperform supervised and semi-supervised unmixing. The package can be found at: https://github.com/BehnoodRasti/HySUPP.

Object detection in remote sensing images (RSIs) plays an important role both in civil and military fields. Currently, many object detection algorithms in RSIs have shown the excellent capability. However, these methods are designed for the single RGB modality, which cannot cope with the challenges in insufficient illumination or foggy scenarios. Infrared images measure the temperature of the captured objects, and it can avoid the influence of low illumination and fog. In this paper, we propose a novel RGB-Infrared multi-modal remote sensing object detection method termed as RIFuse to address these challenges. RIFuse combines convolutional neural networks (CNNs) and Transformer in a parallel hierarchy, which can efficiently extract the local features of RGB images and the global representations of infrared images. Besides, an adaptive multi-modal feature fusion block (MFF block) is proposed to fuse the features from both branches comprehensively. Extensive experiments demonstrate the superiority of our method for multi-modal object detection on RSIs.

Classification using multi-source remote sensing data has attracted widespread attention and is playing an increasingly important role in various fields. However, due to the disparity in imaging methods and the informational imbalance among data from various sources, it is still challenging to incorporate complementary advantages. Furthermore, real-world situations frequently involve varied data resolution, and the outcomes of straightforward super-resolution preprocessing are not always helpful for performing subsequent classification tasks. In this paper, we propose a unified framework for joint super-resolution and classification tasks of low resolution hyperspectral images (LR-HSIs) and SAR images. The minimization of the proposed joint loss function in the generative adversarial network (GAN) framework, including super-resolution and classification objective functions, can effectively achieve the multi-task goals. Experimental results on the super-resolution and classification of real LR-HSIs and SAR images demonstrate the effectiveness of the proposed method both visually and quantitatively.

Recently, convolutional neural networks (CNNs) have been developed for remote sensing image fusion (RSIF). To obtain competitive fusion performance, network design becomes more complicated by stacking convolutional layers deeper and wider. However, problems still remain when applying existing networks in practical applications. On the one hand, researchers focus on improving spatial resolution but ignore that the fused images will be used in subsequent interpretation applications, e.g., objection detection. On the other hand, RSIF involves different tasks with different image sources e.g., pansharpening of the panchromatic and multispectral image, hypersharpening of the panchromatic and hyperspectral image, etc. However, existing networks only solve one of them, failing to be compatible with other tasks. To address the above problems, a convenient task-inspired multiscale nonlocal-attention network (MNAN) is proposed for RSIF. The proposed MNAN focuses more on enhancing the multi-scale targets in the scene when improving the resolution of the fused image. In addition, the proposed network can be applied to both pansharpening and hypersharpening tasks without any modification.

This letter introduces a new sparse unmixing technique using archetypal analysis (SUnAA). First, we design a new model based on archetypal analysis (AA). We assume that the endmembers of interest are a convex combination of endmembers provided by a spectral library and that the number of endmembers of interest is known. Then, we propose a minimization problem. Unlike most conventional sparse unmixing methods, here the minimization problem is nonconvex. We minimize the optimization objective iteratively using an active set algorithm. Our method is robust to the initialization and only requires the number of endmembers of interest. SUnAA is evaluated using two simulated datasets for which results confirm its better performance over other conventional and advanced techniques in terms of signal-to-reconstruction error (SRE). SUnAA is also applied to Cuprite dataset and the results are compared visually with the available geological map provided for this dataset. The qualitative assessment demonstrates the successful estimation of the minerals abundances and significantly improves the detection of dominant minerals compared to the conventional regression-based sparse unmixing methods. The Python implementation of SUnAA can be found at:https://github.com/BehnoodRasti/SUnAA.

In recent years, hyperspectral image classification (HSIC) has achieved impressive progress with emerging studies on deep learning models. However, the classification performance downgrades due to the limited number of annotated samples, especially for minority classes. Notably, the imbalanced data dilemma is familiar in remote sensing hyperspectral image because the ground objects are commonly distributed without evenness. Therefore, this paper proposes a novel deep generative spectral-spatial classifier (DGSSC) for addressing the issues of imbalanced HSIC. Specifically, the DGSSC comprises three components, a two-stage encoder, a decoder, and a classifier, which are trained in an end-to-end manner. In particular, to exploit the abundant spectral-spatial features with relatively low computational complexity, the first stage of the encoder comprises successive three-dimensional (3D) and two-dimensional (2D) convolutions, exploring the spectral-spatial and deep spatial information. In addition, the second stage involves the deep latent variable model to achieve minority-class data augmentation. Furthermore, a patch distance-based reconstruction loss function is designed to facilitate the outputs of the decoder being more similar to the input 3D patch samples. The proposed DGSSC can outperform the state-of-the-art methods on three benchmark datasets, especially with its more robust prediction results. For instance, the DGSSC achieves a remarkable 97.85% mean overall accuracy with 0.24% standard deviation over ten independent runs with randomly selected imbalanced 1% training samples on the University of Pavia dataset.

Hyperspectral image super-resolution (HISR) is about fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to generate a high-resolution hyperspectral image (HR-HSI). Recently, convolutional neural network (CNN)-based techniques have been extensively investigated for HISR yielding competitive outcomes. However, existing CNN-based methods often require a huge amount of network parameters leading to a heavy computational burden, thus, limiting the generalization ability. In this article, we fully consider the characteristic of the HISR, proposing a general CNN fusion framework with high-resolution guidance, called GuidedNet. This framework consists of two branches, including 1) the high-resolution guidance branch (HGB) that can decompose the high-resolution guidance image into several scales and 2) the feature reconstruction branch (FRB) that takes the low-resolution image and the multiscaled high-resolution guidance images from the HGB to reconstruct the high-resolution fused image. GuidedNet can effectively predict the high-resolution residual details that are added to the upsampled HSI to simultaneously improve spatial quality and preserve spectral information. The proposed framework is implemented using recursive and progressive strategies, which can promote high performance with a significant network parameter reduction, even ensuring network stability by supervising several intermediate outputs. Additionally, the proposed approach is also suitable for other resolution enhancement tasks, such as remote sensing pansharpening and single-image super-resolution (SISR). Extensive experiments on simulated and real datasets demonstrate that the proposed framework generates state-of-the-art outcomes for several applications (i.e., HISR, pansharpening, and SISR). Finally, an ablation study and more discussions assessing, for example, the network generalization, the low computational cost, and the fewer network parameters, are provided to the readers. The code link is: https://github.com/Evangelion09/GuidedNet.

Recently, low-rank representation (LRR) methods have been widely applied for hyperspectral anomaly detection, due to their potentials in separating the backgrounds and anomalies. However, existing LRR models generally convert 3-D hyperspectral images (HSIs) into 2-D matrices, inevitably leading to the destruction of intrinsic 3-D structure properties in HSIs. To this end, we propose a novel tensor low-rank and sparse representation (TLRSR) method for hyperspectral anomaly detection. A 3-D TLR model is expanded to separate the LR background part represented by a tensorial background dictionary and corresponding coefficients. This representation characterizes the multiple subspace property of the complex LR background. Based on the weighted tensor nuclear norm and the$L_{F,1}$sparse norm, a dictionary is designed to make its atoms more relevant to the background. Moreover, a principal component analysis (PCA) method can be assigned as one preprocessing step to exact a subset of HSI bands, retaining enough the HSI object information and reducing computational time of the postprocessing tensorial operations. The proposed model is efficiently solved by the well-designed alternating direction method of multipliers (ADMMs). A comparison with the existing algorithms via experiments establishes the competitiveness of the proposed method with the state-of-the-art competitors in the hyperspectral anomaly detection task.

Hyperspectral unmixing is an effective tool to ascertain the material composition of each pixel in a hyperspectral image with typically hundreds of spectral channels. In this paper, we propose two graph-based semi-supervised unmixing methods. The first one directly applies graph learning to the unmixing problem, while the second one solves an optimization problem that combines the linear unmixing model and a graph-based regularization term. Following a semi-supervised framework, our methods require a very small number of training pixels that can be selected by a graph-based active learning method. We assume to obtain the ground truth information at these selected pixels, which can be either the exact abundance value or the one-hot pseudo label. In practice, the latter is much easier to obtain, which can be achieved by minimally involving a human in the loop. Compared to other popular blind unmixing methods, our methods significantly improve performance with minimal supervision. Specifically, the experiments demonstrate that the proposed methods improve the state-of-the-art blind unmixing approaches by 50% or more using only 0.4% of training pixels.

Graph convolutional network (GCN) has recently received increasing attention in hyperspectral image (HSI) classification, benefiting from its superiority in conducting shape adaptive convolutions on arbitrary non-Euclidean structure data. However, the performance of GCN heavily depends on the quality of the initial graph. Conventional GCN-based methods only adopt spectral-spatial similarity to build the initial graph without extracting other contextual information from neighboring nodes. In addition, most GCN-based methods use shallow layers, which cannot extract deep discriminative features from HSIs under the limited number of training samples. To solve these issues, we propose a superpixel feature learning via offset graph U-Net for HSI classification, which can learn deep discriminative features from HSIs. Multiple strategies of measuring similarity among superpixels are utilized to build the initial graph, including spectral information, spatial information and context-aware information among nodes, making the initial graph more accurate. Furthermore, the graph U-Net structure, containing the graph pooling layer and the graph unpooling layer, is helpful in constructing deep GCN layers and learning multi-scale features, which can alleviate the oversmoothing problem. Moreover, an offset module is introduced to emphasize the local spectral-spatial information. Finally, we comprehensively evaluate the proposed method on three public data sets. The experimental results demonstrate the superiority of the proposed approach compared with other state-of-the-art methods.

Deep learning (DL) methods have achieved impressive performance for pansharpening in recent years. However, because of poor generalization, most DL methods achieve unsatisfactory performance for data acquired by sensors not considered during the training phase and decreased performance for samples at full resolution. To solve this issue, we propose a data augmentation framework for pansharpening neural networks. Specifically, we introduce first a random spatial degradation based on anisotropic Gaussian-shaped modulation transfer functions (MTFs) to increase the generalization with respect to different spatial models and sensors. Then, considering that various sensors have different ground sampling distances (GSDs), we randomly rescale the GSD of the training samples to improve the generalization with respect to spatial resolution. Thanks to this module, the generalization to tests from different sensors and samples at full resolution can easily be achieved. Experimental results demonstrate the effectiveness of the proposed approach with better performance when data for training are decoupled with the ones for testing and comparable performance when training and testing are coupled (i.e., data acquired by the same sensor are considered in the two phases). Besides, performance at full resolution for pansharpening neural networks is improved by the proposed approach. The proposed approach has been integrated into existing pansharpening neural networks showing satisfactory performance for widely used sensors, including, GaoFen-1, QuickBird, WorldView-2, WorldView-3, IKONOS, Spot-7, GeoEye, and PHR1A.

The aerial objects tend to distribute with a major variation on the scale and arbitrary orientations in remote sensing images. To meet such characteristics of the aerial object, most of the existing anchor-based detectors rely on preset anchors with variable scales, angles, and aspect ratios, which leads to the misalignment of the selection of candidate regions, the extraction of object features, and the label assignment of preset boxes, interfering the performance of the detector. To address this issue, we propose a Hierarchical Adaptive Alignment Network (HAA-Net). Specifically, we first design the Region Refinement Module (RRM), Feature Alignment module (FAM), and Potential Label Assignment Module (PLAM) to alleviate the misalignment of the region, feature, and label levels respectively; furthermore, we use the gradient equalization strategy to jointly optimize these modules at different levels, so that the whole network can be fully trained to significantly improve detection performance. Extensive experiments demonstrate that our approach can achieve superior performance in three common aerial object datasets (e.g., DOTA, HRSC2016, and UCAS-AOD) when compared with state-of-the-art detectors.

Hyperspectral (HS) pansharpening seeks to integrate low spatial resolution HS (LRHS) images with connected panchromatic (PAN) images to produce high spatial resolution HS (HRHS) images. Traditional pansharpening convolutional neural networks (CNNs) directly map LRHS and PAN images into HRHS images under fixed network parameters, which imply static pansharpening rules. However, real-world HS data are often characterized by spatial variations, and intuitively, the pansharpening rules should be dynamic. To deal with the dilemma, in this article, we develop dynamic HS pansharpening CNNs. We first specify the concepts of dynamic pansharpening and static pansharpening. Then, we propose a learn-to-learn-oriented pansharpening CNN paradigm, which aims to learn a how-to-learn rule to produce spatially adaptive pansharpening rules and comprises three stages of preliminary fusion, scene-sensitive modulation, and spectral reconstruction. Finally, following the paradigm, we design two groups of dynamic pansharpening CNNs (DyPNNs), i.e., internal-connection-based and external-connection-based. They involve various spatial modulations, including spatial affine transform (AT), spatial dynamic convolution (DC), or improved spatial attention (SA), and, thus, consist of six specific DyPNNs: IC-AT-DyPNN, IC-DC-DyPNN, IC-SA-DyPNN, EC-AT-DyPNN, EC-DC-DyPNN, and EC-SA-DyPNN. Experimental results on several HS datasets verify the effectiveness of the proposed DyPNNs in terms of both the spatial reconstruction and spectral fidelity.

Enormous efforts have been recently made to super-resolve hyperspectral (HS) images with the aid of high spatial resolution multispectral (MS) images. Most prior works usually perform the fusion task by means of multifarious pixel-level priors. Yet the intrinsic effects of a large distribution gap between HS-MS data due to differences in the spatial and spectral resolution are less investigated. The gap might be caused by unknown sensor-specific properties or highly-mixed spectral information within one pixel (due to low spatial resolution). To this end, we propose a subpixel-level HS super-resolution framework by devising a novel decoupled-and-coupled network, called DC-Net, to progressively fuse HS-MS information from the pixel- to subpixel-level, from the image- to feature-level. As the name suggests, DC-Net first decouples the input into common (or cross-sensor) and sensor-specific components to eliminate the gap between HS-MS images before further fusion, and then thoroughly blends them by a model-guided coupled spectral unmixing (CSU) net. More significantly, we append a self-supervised learning module behind the CSU net by guaranteeing material consistency to enhance the detailed appearance of the restored HS product. Extensive experimental results show the superiority of our method both visually and quantitatively and achieve a significant improvement in comparison with the state-of-the-art.

Road connectivity is vital in road extraction for accurate vehicle navigation. However, the segmentation-based methods fail to model the connectivity resulting in broken road segments. Therefore, we propose a Location-Guided Network (LGNet) for promoting connectivity performance in a very effective and efficient way. Specifically, an auxiliary Road Location Prediction (RLP) task is designed to obtain global road connectivity information, which improves the performance of road segmentation. The RLP can predict the location coordinates of the whole roads with row anchors and column anchors. By aggregating the global location context to the segmentation branch with a location-guided decoder (LG-Decoder), the features can finally capture the connectivity of each road segment. Overall, LGNet has the following advantages: 1) The proposed RLP and LCG can plug into any encoder-decoder network and achieve an impressive performance. 2) High computational efficiency. In comparison with the multi-branch method, our proposed LGNet requires about 6× fewer GFLOPs. 3) The superior road connectivity performance. A series of experiments are conducted on two road extraction data sets (SpaceNet and DeepGlobe), confirming the effectiveness of the LGNet.

Subspace clustering (SC) has achieved remarkable success in hyperspectral images (HSIs) due to the powerful representation ability of handling high-dimensional complex data. However, most of the existing SC methods focus on linear subspace representation and ignore the more effective nonlinear representation. Besides, SC suffers from the bottlenecks, such as high computation load and memory capacity, due to the spectral decomposition of adjacency matrix for large HSIs. To overcome these limitations, we propose an end-to-end learnable network framework for large HSIs, called self-supervised deep multi-level subspace clustering (S2DMSC), which incorporates the convolutional neural network (CNN) module, multi-level subspace clustering (MSC) module, and high-quality pseudo-label-based self-supervised learning module into a unified learning framework. More concretely, the deep multi-level spatial-spectral representation from hierarchical superpixels is modeled as a sparsity-constrained self-expression module for SC to construct high-quality pseudo-labels to learn the network parameters and produce better clusters for hyperspectral pixels. Experimental results on four classical HSIs demonstrate the effectiveness of S2DMSC and exhibit superior clustering performance compared to the representative clustering methods.

Considerable endeavors have been expended towards enhancing the representation performance for Hyperspectral Anomaly Detection (HAD) through physical model-based methods and recent deep learning-based approaches. Of these methods, the Low-Rank Representation (LRR) model is widely adopted for its formidable separation capabilities for background and target features, however, its practical applications are limited due to the reliance on manual parameter selection and subpar generalization performance. To this end, this paper presents a new HAD baseline network, referred to as LRR-Net, which synergizes the LRR model with deep learning techniques. LRR-Net leverages the alternating direction method of multipliers (ADMM) optimizer to solve the LRR model efficiently and incorporates the solution as prior knowledge into the deep network to guide the optimization of parameters. Moreover, LRR-Net transforms the regularized parameters into trainable parameters of the deep neural network, thus alleviating the need for manual parameter tuning. Additionally, this paper proposes a sparse neural network embedding to demonstrate the scalability of the LRR-Net framework. Empirical evaluations on eight distinct datasets illustrate the efficacy and superiority of the proposed approach compared to state-of-the-art methods.

For unmixing of sequences of hyperspectral images (SHS), spectral variability is an important factor to be considered. However, most existing unmixing methods tend to model the endmember and its variability in spatial domain rather than transform domain. In fact, the intrinsic and invariant features of the spectral curve can be effectively represented by wavelet transform. Therefore, this paper proposes to perform SHS unmixing in the wavelet domain by combing the Bayesian method. Firstly, the assumption of abundance being invariability in both the spatial and wavelet domains is made, then the formulation of unmixing in the wavelet domain using Perturbed Linear Mixing Model (PLMM) is presented. Secondly, based on the Bayesian framework, the likelihood and prior are both given, in which the parameter priors are divided into two parts: low and high frequency wavelet coefficients. Moreover, by considering the sparsity of the high-frequency wavelet coefficients of endmembers, a non-informative prior with zero-mean is designed. Meanwhile, for the coefficients of endmember variability, Gaussian distributions are utilized to represent the steady fluctuation along the temporal dimension. Finally, using the maximum a posterior (MAP) rule, a hierarchical spectral variability unmixing model in wavelet domain is built and solved by the Markov chain Monte Carlo (MCMC) sampling algorithm. Numerical experiments show that the proposed method generates more accurate estimates for endmembers and their variation.

Deep learning-based semantic segmentation has made great progress in understanding very-high-resolution (VHR) remote sensing images (RSIs). However, large-scale applications are still limited. The main reason is that diverse imaging modes and geographical differences make it difficult to transfer a model trained in the source domain to the target domain. To solve this problem, unsupervised domain adaptation (UDA) for VHR RSIs has received some attention, but the accuracy of cross-domain semantic segmentation still needs to be improved. Currently, one reasonable proposal for improving accuracy is to take a close look at the category-level information. In this paper, we reveal an integer programming mechanism for modeling the category-level relationship between the source and target domains. The mechanism is based on the solution of the assignment problem, and thus, the proposed method is called category-level assignment for UDA (ClA-UDA). In ClA-UDA, a category-level assignment problem with additional constraints is defined for UDA tasks, and the solution is provided. Based on the solution, an assignment-based image-to-image transferring algorithm (AIT) is first proposed to transfer the source-domain images based on the style of the target-domain images. AIT minimizes a weighted discrepancy, and provides an analytical solution for the transfer. Two assignment-based alignment losses are then introduced to align the source and target domains based on the category-level relationship in a concise way. To validate the performance of ClA-UDA, three VHR remote sensing image datasets are employed, and six UDA tasks are designed. Extensive experiments are conducted, and the results demonstrate the superiority of ClA-UDA compared to the existing methods.

International audience

Hyperspectral unmixing aims at estimating pure spectral signatures and their proportions in each pixel. In practice, the atmospheric effects, intrinsic variation of the spectral signatures of the materials, illumination, and topographic changes cause what is known as spectral variability resulting in significant estimation errors being propagated throughout the unmixing task. To this end, we developed a new method, called the orthogonal subspace unmixing (OSU), to address spectral variability by utilizing the orthogonal subspace projection. The proposed OSU method jointly performs orthogonal subspace learning and the unmixing process to find a more suitable subspace for unmixing. The orthogonal subspace projection encourages the representation held in the subspace to be more distinct from each other to remove the complex spectral variability in the subspace. Furthermore, an alternating minimization (AM) was designed to solve the resulting optimization problem. An efficient and convergent symmetric Gauss–Seidel alternating direction method of multipliers (sGS-ADMM), essentially a special case of the semiproximal alternating direction method of multipliers (SPADMM), was developed to solve the subproblem. Experiments conducted on one synthetic data and two real data demonstrate the effectiveness and superiority of the proposed framework in mitigating the effects of spectral variability with respect to classical linear unmixing methods or variability accounting approaches.

During incremental learning tasks, catastrophic forgetting occurs when old models are updated with new information. To address this issue, we propose a novel method called label strategy and adaptive weights (LSAW) that improves the incremental learning process. The label strategy introduces the old classes and solves the problem of how to reasonably use the wrong samples predicted by the old model. In the cross-entropy (CE) loss, we apply a threshold to filter the pseudolabels predicted by the old model. Subsequently, we merge the pixel samples with high probability with the current label. The probability here refers to the probability that the pixel belongs to the true class. This process enables the introduction of information from old classes that are not directly accessible in the current stage. Moreover, this information is relatively reliable, and the model exhibits confidence in its accuracy. For the remaining pixels, we retain all classes’ information through label smoothing. In the distillation function, the old class and background pixel samples are selected for distillation according to the prediction map of the old classes. The weights of the classes are adaptively updated and adjusted using specific label information from each batch and the different stages of incremental learning. As demonstrated by the results of our experiment, on three remote sensing image datasets: China Computer Federation (CCF), Potsdam, and Vaihingen, our method achieves the best results.

Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared to convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR) offer the potential to improve these models by means of a CLS. In this paper, we introduce a new multimodal fusion transformer (MFT) network which comprises a multihead cross patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a distinctive representation in a reduced and hierarchical feature space. Extensive experiments are carried out on widely used benchmark datasets i.e., the University of Houston, Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of multihead cross patch attention. The source code will be made available publicly at https://github.com/AnkurDeria/MFT.

Efficient object detection methods have recently received great attention in remote sensing. Although deep convolutional networks often have excellent detection accuracy, their deployment on resource-limited edge devices is difficult. Knowledge distillation (KD) is a strategy for addressing this issue since it makes models lightweight while maintaining accuracy. However, existing KD methods for object detection have encountered two constraints. First, they discard potentially important background information and only distill nearby foreground regions. Second, they only rely on the global context, which limits the student detector’s ability to acquire local information from the teacher detector. To address the aforementioned challenges, we propose Attention-based Feature Distillation (AFD), a new KD approach that distills both local and global information from the teacher detector. To enhance local distillation, we introduce a multi-instance attention mechanism that effectively distinguishes between background and foreground elements. This approach prompts the student detector to focus on the pertinent channels and pixels, as identified by the teacher detector. Local distillation lacks global information, thus attention global distillation is proposed to reconstruct the relationship between various pixels and pass it from teacher to student detector. The performance of AFD is evaluated on two public aerial image benchmarks, and the evaluation results demonstrate that AFD in object detection can attain the performance of other state-of-the-art models while being efficient.

The injection scheme provides an efficient way for CS- and MRA-based pansharpening approaches. Within this paradigm, the estimation of injection gains is one of the keys to pansharpening outcomes, which has attracted much attention in the community. Most of the existing models are derived from the regression methodology. Hence, the reference is indispensable for the estimation. However, the reference is unavailable in practice, and therefore, the estimation is usually performed at a degraded scale. This article is devoted to the estimation of injection gains without reference. A hybrid-scale (HS) estimation, which involves both the high-resolution and low-resolution data, is proposed, along with three HS models. The proposed method features a context-based and fast implementation with fewer tunable parameters. Experimental results show that the HS models yield more accurate and robust results compared with the typical regression-based models, and they are also competitive with the state-of-the-art approaches.

Efforts to enhance the detection accuracy of hyperspectral (HS) anomaly detection (AD) have been significant, but the impact of noise resulting from HS data acquisition and transmission has not been well studied. Furthermore, the separation of denoising and subsequent interpretation makes it challenging to evaluate and control the influence of noise on the detection results. To this end, we proposed a joint anomaly detection and noise removal (ADNR) paradigm called DSR-ADNR, which develops a double subspace representation method to obtain both denoised and detection results simultaneously. DSR-ADNR uses a low-dimensional orthogonal basis to represent HS images and extract distinctive features for AD. The feature matrix is represented by a dictionary-based low-rank subspace that captures the complex nature of the low-dimensional features. In each iteration, DSR-ADNR utilizes the nonlocal self-similarity of the feature matrix to remove noise and improve intermediate detection performance. Meanwhile, the progressive LR representation of the background and anomalies for the feature matrix upgrades the explicit LR expression of nonlocal self-similar patches for better denoising. The well-designed linearized alternating direction method of multipliers with an adaptive penalty (LADMAP) is utilized to solve the proposed DSR-ADNR. Extensive experiments on simulated and real-world data sets demonstrate the effectiveness of DSR-ADNR in the HS AD task under different noise cases.

Remote sensing scene understanding is a highly challenging task, and has gradually emerged as a research hotspot in the field of intelligent interpretation of remote sensing data. Recently, the use of convolutional neural networks (CNNs) has been proven to be a fruitful advancement. However, with the emergence of visual transformers (ViTs), the limitations of traditional small convolutional kernels in directly capturing a large receptive field have posed significant challenges to their dominant role. Additionally, the fixed neuron connections between different convolutional layers have weakened the practicality and adaptability of the models. Furthermore, the global average pooling also leads to the loss of effective information in the acquired features. In this work, a Large kernel Sparse ConvNet weighted by Multi-frequency Attention (LSCNet) is proposed. Firstly, unlike traditional convolutional neural networks, it utilizes two parallel rectangular convolutional kernels to approximate a large kernel, achieving comparable or even better results than ViTs-based methods. Secondly, an adaptive sparse optimization strategy is employed to dynamically optimize the fixed neuron connections between different convolutional layers, achieving a favorable connectivity pattern for capturing abstract features more accurately. Lastly, a novel multi-frequency attention (MFA) module is used to replace global average pooling (GAP), so as to preserve more useful information while weighting the recognition features, thereby enhancing the discriminative and learning abilities of the model. In the conducted experiments, LSCNet achieves the best recognition results on three well-known remote sensing aerial datasets when compared to the state-of-the-art methods (including ViTs-based methods).

In recent years, remote sensing scene classification is one of research hotspots and has played an important role in the field of intelligent interpretation of remote sensing data. However, various complex objects and backgrounds form a variety of remote sensing scenes through spatial combination and correlation, which brings great challenges to accurately classify different scenes. Among them, the insufficient feature difference brought about the unbalanced change of background and target between inter-class sample and the feature representation inconsistency caused by the difference of representation among the intra-class samples have become obstacles to effectively distinguish different scene images. To address these issues, a Multi-stage Self-Guided Separation Network (MGSNet) is proposed for remote sensing scene classification. First of all, different from the previous work, it attempts to utilize the background information outside the effective target in the image as a decision aid through a target-background separation strategy to improve the distinguish ability between target similarity-background difference samples. In addition, the diversity of feature concerns among different network branches is expanded through contrastive regularization to improve the separation of target-background information. Additionally, a self-guided network is proposed to find common features between intra-class samples and improve the consistency of feature representation. It combines the texture and morphological features of images to guide feature learning, effectively reducing the impact of intra-class differences. Extensive experimental results on three benchmark demonstrate that MGSNet can achieve better classification performance compared to the state-of-the-art approaches.

Poisson noise is one of the significant sources of noise present in hyperspectral imagery (HSI). In most of the existing denoising methods, Poisson noise is first transformed into Gaussian noise through the Anscombe transform and then removed. However, the use of Anscombe transform can give rise to transform errors that affect the final denoising results. In addition, blurs often contaminate the HSI during the imaging procedure, which makes it more difficult to remove the Poisson noise. In view of the above problems, under the maximum a posteriori (MAP) model, we propose a Poissonian blurred HSI denoising based on variable splitting and penalty technique (named as VSPT) to directly remove the Poissonian blurred HSI noise without using the Anscombe transform. By finding the minimum value of the negative logarithmic Poisson log-likelihood combined with the total variation (TV), the proposed method transforms the problem into two subproblems, which are easier to solve: 1) a TV regularized deconvolution problem and 2) an ordinary convex optimization problem. The experimental results show that the proposed VSPT method can effectively remove Poisson noise in HSI contaminated by blurs during the imaging procedure.

The emerging research line of cross-modal learning focuses on the issue of transferring feature representation manner learned from limited multimodal data with labelings to the testing phase with partial modalities. This is essentially common and practical in the remote sensing community when only modal-incomplete data are in users’ hands due to inevitable imaging or access restrictions under large-scale observation scenarios. However, most of the existing cross-modal learning methods have been designed with exclusive reliance on labeling, which can be either limited or noisy due to their costly production. To address this issue, we explore in this paper the possibility to learn cross-modal feature representation in an unsupervised fashion. By integrating the multimodal data into a fully recombined matrix form, we propose 1) the use of common subspace representation as the regression target instead of conventionally adopted binary labels, and 2) the orthogonality and manifold alignment regularization terms to shrink the solution space whilst preserving the pairwise manifold correlations. Through this manner, the modality-specific and mutual latent representations in this common subspace as well as their corresponding projections can be learned simultaneously and their optimums can be efficiently reached through a nearly one-step computation with the help of Eigen decomposition. Finally, we show the superiority of our method through extensive image classification experiments on three multimodal datasets with four remotely sensed modalities involved (i.e., hyperspectral, multispectral, synthetic aperture radar, and light detection and ranging data). The code and dataset will be made freely available at https://github.com/jingyao16/UCSL after a possible publication to encourage the reproduction of our method and further use.

The recent success of attention mechanism-driven deep models, like Vision Transformer (ViT) as one of the most representative, has intrigued a wave of advanced research to explore their adaptation to broader domains. However, current Transformer-based approaches in the remote sensing (RS) community pay more attention to single-modality data, which might lose expandability in making full use of the ever-growing multimodal Earth observation data. To this end, we propose a novel multimodal deep learning framework by extending conventional ViT with minimal modifications, abbreviated as ExViT, aiming at the task of land use and land cover classification. Unlike common stems that adopt either linear patch projection or deep regional embedder, our approach processes multimodal RS image patches with parallel branches of position-shared ViTs extended with separable convolution modules, which offers an economical solution to leverage both spatial and modality-specific channel information. Furthermore, to promote information exchange across heterogeneous modalities, their tokenized embeddings are then fused through a cross-modality attention module by exploiting pixel-level spatial correlation in RS scenes. Both of these modifications significantly improve the discriminative ability of classification tokens in each modality and thus further performance increase can be finally attained by a full tokens-based decision-level fusion module. We conduct extensive experiments on two multimodal RS benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral and light detection and ranging (LiDAR) data, and Berlin dataset with hyperspectral and synthetic aperture radar (SAR) data, to demonstrate that our ExViT outperforms concurrent competitors based on Transformer or convolutional neural network (CNN) backbones, in addition to several competitive machine learning-based models. The source codes and investigated datasets of this work will be made publicly available at https://github.com/jingyao16/ExViT.

Learning-based infrared small object detection methods currently rely heavily on the classification backbone network. This tends to result in tiny object loss and feature distinguishability limitations as the network depth increases. Furthermore, small objects in infrared images are frequently emerged bright and dark, posing severe demands for obtaining precise object contrast information. For this reason, we in this paper propose a simple and effective "U-Net in U-Net" framework, UIU-Net for short, and detect small objects in infrared images. As the name suggests, UIU-Net embeds a tiny U-Net into a larger U-Net backbone, enabling the multi-level and multi-scale representation learning of objects. Moreover, UIU-Net can be trained from scratch, and the learned features can enhance global and local contrast information effectively. More specifically, the UIU-Net model is divided into two modules: the resolution-maintenance deep supervision (RM-DS) module and the interactive-cross attention (IC-A) module. RM-DS integrates Residual U-blocks into a deep supervision network to generate deep multi-scale resolution-maintenance features while learning global context information. Further, IC-A encodes the local context information between the low-level details and high-level semantic features. Extensive experiments conducted on two infrared single-frame image datasets, i.e., SIRST and Synthetic datasets, show the effectiveness and superiority of the proposed UIU-Net in comparison with several state-of-the-art infrared small object detection methods. The proposed UIU-Net also produces powerful generalization performance for video sequence infrared small object datasets, e.g., ATR ground/air video sequence dataset. The codes of this work are available openly at https://github.com/danfenghong/IEEE.

Pansharpening refers to the fusion of a low spatial-resolution multispectral image with a high spatial-resolution panchromatic image. In this paper, we propose a novel low-rank tensor completion (LRTC)-based framework with some regularizers for multispectral image pansharpening, called LRTCFPan. The tensor completion technique is commonly used for image recovery, but it cannot directly perform the pansharpening or, more generally, the super-resolution problem because of the formulation gap. Different from previous variational methods, we first formulate a pioneering image super-resolution (ISR) degradation model, which equivalently removes the downsampling operator and transforms the tensor completion framework. Under such a framework, the original pansharpening problem is realized by the LRTC-based technique with some deblurring regularizers. From the perspective of regularizer, we further explore a local-similarity-based dynamic detail mapping (DDM) term to more accurately capture the spatial content of the panchromatic image. Moreover, the low-tubal-rank property of multispectral images is investigated, and the low-tubal-rank prior is introduced for better completion and global characterization. To solve the proposed LRTCFPan model, we develop an alternating direction method of multipliers (ADMM)-based algorithm. Comprehensive experiments at reduced-resolution (i.e., simulated) and full-resolution (i.e., real) data exhibit that the LRTCFPan method significantly outperforms other state-of-the-art pansharpening methods. The code is publicly available at: https://github.com/zhongchengwu/code_LRTCFPan.

The synthesis of high-resolution (HR) hyperspectral image (HSI) by fusing a low-resolution HSI with a corresponding HR multispectral image has emerged as a prevalent HSI super-resolution (HSR) scheme. Recent researches have revealed that tensor analysis is an emerging tool for HSR. However, most off-the-shelf tensor-based HSR algorithms tend to encounter challenges in rank determination and modeling capacity. To address these issues, we construct nonlocal patch tensors (NPTs) and characterize low-rank structures with coupled Bayesian tensor factorization. It is worth emphasizing that the intrinsic global spectral correlation and nonlocal spatial similarity can be simultaneously explored under the proposed model. Moreover, benefiting from the technique of automatic relevance determination, we propose a hierarchical probabilistic framework based on Canonical Polyadic (CP) factorization, which incorporates a sparsity-inducing prior over the underlying factor matrices. We further develop an effective expectation-maximization-type optimization scheme for framework estimation. In contrast to existing works, the proposed model can infer the latent CP rank of NPT adaptively without tuning parameters. Extensive experiments on synthesized and real datasets illustrate the intrinsic capability of our model in rank determination as well as its superiority in fusion performance.

In this paper, we introduce a new algorithm based on archetypal analysis for blind hyperspectral unmixing, assuming linear mixing of endmembers. Archetypal analysis is a natural formulation for this task. This method does not require the presence of pure pixels (i.e., pixels containing a single material) but instead represents endmembers as convex combinations of a few pixels present in the original hyperspectral image. Our approach leverages an entropic gradient descent strategy, which (i) provides better solutions for hyperspectral unmixing than traditional archetypal analysis algorithms, and (ii) leads to efficient GPU implementations. Since running a single instance of our algorithm is fast, we also propose an ensembling mechanism along with an appropriate model selection procedure that make our method robust to hyper-parameter choices while keeping the computational complexity reasonable. By using six standard real datasets, we show that our approach outperforms state-of-the-art matrix factorization and recent deep learning methods. We also provide an open-source PyTorch implementation: https://github.com/inria-thoth/EDAA.

In practice, the acquirement of labeled samples for hyperspectral image (HSI) is time-consuming and labor-intensive. It frequently induces the trouble of model overfitting and performance degradation for the supervised methodologies in HSI classification (HSIC). Fortunately, semisupervised learning can alleviate this deficiency, and graph convolutional network (GCN) is one of the most effective semisupervised approaches, which propagates the node information from each other in a transductive manner. In this study, we propose a cross-scale graph prototypical network (X-GPN) to achieve semisupervised high-quality HSIC. Specifically, considering the multiscale appearance of the land covers in the same remotely captured scene, we involve the neighborhoods of different scales to construct the adjacency matrices and simultaneously design a multibranch framework to investigate the abundant spectral-spatial features through graph convolutions. Furthermore, to exploit the complementary information between different scales, we simply employ the standard 1-D convolution to excavate the dependence of the intranode and concatenate the output with the features generated from other scales. Intuitively, different branches for various samples should have different importance to predict their categories. Thus, we develop a self-branch attentional addition (SBAA) module to adaptively highlight the most critical features produced by multiple branches. In addition, different from previous GCN for HSIC, we devise an innovative prototypical layer comprising a distance-based cross-entropy (DCE) loss function and a novel temporal entropy-based regularizer (TER), which can enhance the discrimination and representativeness of the node features and prototypes actively. Extensive experiments demonstrate that the proposed X-GPN is superior to the classic and state-of-the-art (SOTA) methods in terms of the classification performance.

Pansharpening refers to the fusion of a panchromatic (PAN) image with a high spatial resolution and a multispectral (MS) image with a low spatial resolution, aiming to obtain a high spatial resolution MS (HRMS) image. In this article, we propose a novel deep neural network architecture with level-domain-based loss function for pansharpening by taking into account the following double-type structures, i.e., double-level, double-branch, and double-direction, called as triple-double network (TDNet). By using the structure of TDNet, the spatial details of the PAN image can be fully exploited and utilized to progressively inject into the low spatial resolution MS (LRMS) image, thus yielding the high spatial resolution output. The specific network design is motivated by the physical formula of the traditional multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and some multi-scale convolution kernels to deepen and widen the network to effectively enhance the feature extraction and the robustness of the proposed TDNet. Extensive experiments on reduced- and full-resolution datasets acquired by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of the proposed TDNet compared with some recent state-of-the-art pansharpening approaches. An ablation study has also corroborated the effectiveness of the proposed approach. The code is available at https://github.com/liangjiandeng/TDNet.

Multimodal approaches for Earth Observations suffer from both the lack of interpretability of SAR images and the high sensitivity to meteorological conditions of optical images. Translation methods were implemented to solve them for specific tasks and areas. But these implementations lack of generalizability as they do not include samples with challenging characteristics. Firstly, this paper sums up the main problems that a general SAR to optical image translator should overcome. Then, a SAR Distorted Image to optical translator Network (SARDINet) alternating knowledgeable channel-wise spatial convolutions and cross-channel convolutions is implemented. It aims at solving a problem of major concern in remote sensing: translating layover disturbed SAR images into disturbance-free optical ones. SARDINet is trained through a classical and an adversarial framework and compared to cGAN and cycleGAN from the literature. Experimental results prove that adversarial approaches are more qualitative but worsen quantitative results.

Owing to the powerful feature extraction and representation capabilities, deep learning (DL) has been successfully applied in hyperspectral unmixing (HU). However, only relying on hyperspectral data for unmixing fails to distinguish objects with similar spectral information, resulting in the degradation of unmixing performance. To this end, this paper presents a novel multimodal unmixing network, MUNet for short, by considering the height information of light detection and ranging (LiDAR) data in a squeeze-and-excitation (SE) attention fashion to guide the unmixing process toward a more accurate performance. MUNet is capable of efficiently embedding the height information obtained from LiDAR data into the autoencoder unmixing architecture through the attention mechanism, thereby fusing more spatial information to obtain ideal unmixing results. Experimental results conducted on the real multimodal dataset demonstrate the effectiveness and superiority of the proposed MUNet compared to several state-of-the-art deep unmixing approaches.

Over the past few decades, a large collection of feature ex-traction and classification algorithms have been developed for land cover mapping using remote sensing data. Although these methods have shown the gradually-increasing performance, their potential inevitably meets the bottleneck due to the lack of high-quality and diversified remote sensing bench-mark datasets, particularly for the multimodal cases. Accordingly, this, to a larger extent, limits the development of the corresponding methodologies and the practical application of land cover classification. To this end, we aim in this pa-per to introduce and build several multimodal remote sensing benchmark datasets for land cover classification. Further-more, two new multimodal land cover classification bench-mark datasets, i.e., Berlin and Augsburg, are openly available. Experiments are conducted on the two datasets for evaluating the performance of several multimodal feature learning and classification methods.

Convolutional neural networks (CNNs) have achieved impressive performance for hyperspectral (HS) and multispectral (MS) image fusion in recent years. They extract features by local filters, which is limited to explore long-range dependency in input images. However, long-range dependence is an import cue for HS and MS image fusion, as it contributes to exploration of spatial self-similarity and spectral dependence. To take advantage of long-range dependence, we propose a spectral-spatial transformer (SST) for MS and HS image fusion. The experimental results demonstrate the high performance of the proposed approach compared to some state-of-the-art methods.

Deep learning has proven to be a very effective approach for Hyperspectral Image (HSI) classification. However, deep neural networks require large annotated datasets to generalize well. This limits the applicability of deep learning for HSI classification, where manually labelling thousands of pixels for every scene is impractical. In this paper, we propose to leverage Self Supervised Learning (SSL) for HSI classification. We show that by pre-training an encoder on unlabeled pixels using Barlow-Twins, a state-of-the-art SSL algorithm, we can obtain accurate models with a handful of labels. Experimental results demonstrate that this approach significantly outperforms vanilla supervised learning.

Hyperspectral remote sensing image is rich in spectral information. Due to the limitations of sensors and the complexity of the scene, a large number of mixed pixels exist in the scene. Therefore, it is very necessary to develop the unmixing technology. The linear unmixing model and its derived algorithms have made some progress. The existing unmixing methods treat all pixels in the scene as mixed pixels for operation, but the real scene is often a complex scene with pure pixels and mixed pixels. Considering the unmixing of complex scenes, the robust linear unmixing model based on enhanced constraint of classification (ECRLU) is proposed in this paper. The model combines unmixing and classification. After extracting endmembers, the number of endmembers is expanded by using local similarity and spatial similarity to obtain hard classification items, so as to provide sparsity constraints for the model. In this paper, synthetic data set and real data set are used to verify the effectiveness of the model.

This paper proposes a deep blind hyperspectral unmixing network for datasets without pure pixels called minimum simplex convolutional network (MiSiCNet). MiSiCNet is the first deep learning-based blind unmixing method proposed in the literature which incorporates both spatial and geometrical information of the hyperspectral data, in addition to the spectral information. The proposed convolutional encoder-decoder architecture incorporates the spatial information using convolutional filters and implicitly applying a prior on the abundances. We added a minimum simplex volume penalty term to the loss function to exploit the geometrical information. We evaluate the performance of MiSiCNet on simulated and real datasets. The experimental results confirm the robustness of the proposed method to both noise and absence of pure pixels. Additionally, MiSiCNet considerably outperforms the state-of-the-art unmixing approaches. The results are given in terms of spectral angle distance in degree for the endmember estimation, and root mean square error in percentage for the abundance estimation. MiS-iCNet was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online: https://github.com/BehnoodRasti/MiSiCNet.

Small-object detection is a challenging problem. In the last few years, the convolution neural networks methods have been achieved considerable progress. However, the current detectors struggle with effective features extraction for small-scale objects. To address this challenge, we propose image pyramid single-shot detector (IPSSD). In IPSSD, single-shot detector is adopted combined with an image pyramid network to extract semantically strong features for generating candidate regions. The proposed network can enhance the small-scale features from a feature pyramid network. We evaluated the performance of the proposed model on two public datasets and the results show the superior performance of our model compared to the other state-of-the-art object detectors.

Abstract Although hyperspectral data, especially spaceborne images, are rich in spectral information, their spatial resolution is usually low due to the limitation of sensor design and other factors. Therefore, for the application of hyperspectral images, unmixing technology is a key processing technology, such as linear mixing model and its derived algorithms have made a certain progress. However, a real scene often contains both pure and mixed pixels. The existing methods usually ignore the consideration and analysis of this situation in the process of model design and simulation experiment. In this context, this paper proposes a robust linear unmixing model with the enhanced constraint of classification for hyperspectral image. In general, it designs a framework combining unmixing and classification. In the task for real scene data, endmembers are extracted first, and then the hard classification term constructed after the expansion of endmembers (training samples) based on similarity is introduced to provide the sparsity constraint of the overall model, so as to realize relatively complete adjustment and effective image unmixing under complex conditions. Considering the scene with different distributions, the simulation experiment designs several groups of data tests, including different proportions of pure and mixing pixels. The unmixing results of three simulated datasets and two real datasets show that the unmixing results of this method are better than those of the other six comparison methods. This model improves the accuracy of unmixing and realizes effective unmixing.

Polarimetric synthetic aperture radar (PolSAR) image classification has been investigated vigorously in various remote sensing applications. However, it is still a challenging task nowadays. One significant barrier lies in the speckle effect embedded in the PolSAR imaging process, which greatly degrades the quality of the images and further complicates the classification. To this end, we present a novel PolSAR image classification method that removes speckle noise via low-rank (LR) feature extraction and enforces smoothness priors via the Markov random field (MRF). Especially, we employ the mixture of Gaussian-based robust LR matrix factorization to simultaneously extract discriminative features and remove complex noises. Then, a classification map is obtained by applying a convolutional neural network with data augmentation on the extracted features, where local consistency is implicitly involved, and the insufficient label issue is alleviated. Finally, we refine the classification map by MRF to enforce contextual smoothness. We conduct experiments on two benchmark PolSAR data sets. Experimental results indicate that the proposed method achieves promising classification performance and preferable spatial consistency.

In this letter, a novel neural architecture search (NAS) method based on reinforcement learning, called RLNAS, is devised to realize the automatic architecture design in the field of hyperspectral unmixing (HU). This method first train the search network in the constructed self-supervised datasets based on hyperspectral images. The block-based searching and weight-sharing strategies are then introduced to reduce the computational cost in the training phase. The final optimal architecture is obtained by optimizing the multi-objective reward function to balance the trade-off between accuracy and computational efficiency. Compared with the state-of-the-art unmixing algorithms, the proposed RLNAS method can yield better unmixing results on synthetic and real hyperspectral datasets, which verifies its effectiveness and superiority. In addition, the proposed method offers promising potential of the NAS for HU.

Deep learning (DL) has been garnering increasing attention in remote sensing (RS) due to its powerful data representation ability. In particular, deep models have been proven to be effective for RS data classification based on a single given modality. However, with one single modality, the ability in identifying the materials remains limited due to the lack of feature diversity. To overcome this limitation, we present a simple but effective multimodal DL baseline by following a deep encoder–decoder network architecture, EndNet for short, for the classification of hyperspectral and light detection and ranging (LiDAR) data. EndNet fuses the multimodal information by enforcing the fused features to reconstruct the multimodal input in turn. Such a reconstruction strategy is capable of better activating the neurons across modalities compared with some conventional and widely used fusion strategies, e.g., early fusion, middle fusion, and late fusion. Extensive experiments conducted on two popular hyperspectral and LiDAR data sets demonstrate the superiority and effectiveness of the proposed EndNet in comparison with several state-of-the-art baselines in the hyperspectral-LiDAR classification task. The codes will be available athttps://github.com/danfenghong/IEEE_GRSL_EndNet, contributing to the RS community.

Owing to easy acquisition and large coverage from the space, multispectral (MS) imaging has garnered growing interest in various applications of remote sensing. However, the limited spectral information of MS data, to a great extent, leads to difficulties in classifying the materials more accurately, particularly for those classes that have very similar visual appearances. To address this issue effectively, we attempt to enhance the spectral resolution of MS imagery, enabling the identification of materials at a more precise level by the means of richer spectral information. More specifically, we propose to learn locality-constrained sparse coding (LCSC) for short, on partially overlapped hyperspectral (HS)-MS pairs (i.e., dictionary). LCSC is capable of capturing neighboring relations well by enforcing the local constraint for each pixel. Such a strategy makes it possible to better reconstruct HS products from MS images and partially overlapped HS images. Reconstruction and unmixing are explored as potential applications to assess the performance of spectral enhancement. Extensive experiments are conducted on two HS-MS data sets in comparison with several state-of-the-art baselines, which demonstrate the effectiveness of the proposed LCSC algorithm in the task of spectral enhancement.

Due to sensor malfunction and atmosphere disturbances, high-dimensional optical remote sensing (HORS) images often suffer from information missing, such as dead pixels and thick clouds. Tensor decomposition methods have been used to estimate the missing data of HORS images. However, most existing models hardly consider the inherent properties and effective structural information of HORS images. To this end, we propose a novel total variation (TV) regularized weighted tensor ring (TR) decomposition model to recover the missing content of HORS images. The TR decomposition has the powerful low-rank (LR) representation ability to recover the HORS data by employing three low-dimensional tensors, i.e., TR factors. An initialization step and proper weights are designed to enhance the flexibility for exploring different LR properties of TR factors. To further preserve the spatial smoothness, the local spatial TV from three directions is incorporated into the TR decomposition framework. Furthermore, an augmented Lagrange multiplier (ALM) algorithm is designed for solving the resulting optimization problem. Experiments on HORS images demonstrate the performances of the proposed method over the current state-of-the-art baselines.

Infrared objects acquired from a long-distance have small sizes and are easily submerged by a complex and variable background. The existing deep network detection framework suffers greatly from the feature spatial resolution loss caused by the networks’ depth and multiple downsampling operations, which is extremely detrimental for small object detection. So, a crucial and urgent goal is, how to trade-off network depth and feature spatial resolution, while learning feature context representation and interaction to distinguish from the background. To this end, we propose a deep interactive U-Net architecture (short for DI-U-Net) with high feature learning and feature interaction ability. First, feature learning is first achieved through a multi-level and high-resolution network structure. This structure ensures feature resolution as the network depth increase, and also focus on the object’s global context information. Then, the feature interactive is further achieved by the dense feature encoder (DFI) module to learn object local context information. The proposed method yields strong object context representation and well discriminability, as well as a good fit for infrared small object detection. Extensive experiments are conducted on the SISRT dataset and Synthetic dataset, demonstrating the superiority and effectiveness of the proposed deeper U-Net compared to previous state-of-the-art detection methods.

This letter proposes a semi-supervised mixtures of factor analyzers (S2MFA) feature extraction (FE) method for hyperspectral image (HSI). S2MFA uses a Gaussian mixture model to segment the image to different regions, each region follows a Gaussian distribution and contains labeled and unlabeled samples. The method uses a factor analyzer to get a factor-loading matrix to preserve the local spatial information using the labeled and unlabeled samples. It simultaneously improves the class discrimination of the data using the labeled samples and also transforms the original image to an optimal low-dimensional subspace to achieve dimensionality reduction. The performance of the S2MFA FE method is evaluated by classification of two real HSIs and compared to different kinds of statistic unsupervised, supervised, and semi-supervised FE methods.

Traditional hyperspectral (HS) pansharpening aims at fusing a HS image with its panchromatic (PAN) counterpart, to bring the spatial resolution of the HS image to that of the PAN image. However, in many practical applications, arbitrary resolution HS (ARHS) pansharpening is required, where the HS and PAN images need to be integrated to generate a pansharpened HS image with arbitrary resolution (usually higher than that of the PAN image). Such an innovative task brings forth new challenges for the pansharpening technique, mainly including how to reconstruct HS images beyond the training scale and how to guarantee spectral fidelity at any spatial resolutions. To tackle the challenges, we present a novel convolutional neural network (CNN)-based method for ARHS pansharpening called ARHS-CNN. It is based on a two-step relay optimization process, which is associated with a multilevel enhancement subnetwork and a rescaling subnetwork. With a careful design following the thread, our ARHS-CNN is able to pansharpen HS images to any spatial resolutions using just a single CNN model trained on a limited number of scales while meantime to keep spectral fidelity at those resolutions, which wins an obvious advantage over traditional pansharpening methods. Experimental results obtained on several datasets verify the excellent performance of our ARHS-CNN method.

Owing to the powerful and automatic representation capabilities, deep learning (DL) techniques have made significant breakthroughs and progress in hyperspectral unmixing (HU). Among the DL approaches, autoencoders (AEs) have become a widely-used and promising network architecture. However, these AE-based methods heavily rely on manual design and may not be a good fit for specific datasets. To unmix hyperspectral images more intelligently, we propose an automatic neural architecture search model for HU, AutoNAS for short, to determine the optimal network architecture by considering channel configurations and convolution kernels simultaneously. In AutoNAS, the self-supervised training mechanism based on hyperspectral images is first designed for generating the training samples of the supernet. Then, the affine parameter sharing strategy is adopted by applying different affine transformations on the supernet weights in the training phase, which enables finding the optimal channel configuration. Furthermore, on the basis of the obtained channel configuration, the evolutionary algorithm with additional computational constraints is introduced into networks to achieve flexible convolution kernel search by evaluating unmixing results of different architectures in the supernet. Extensive experiments conducted on four hyperspectral datasets demonstrate the effectiveness and superiority of the proposed AutoNAS in comparison with several state-of-the-art unmixing algorithms.

Convolutional Neural Networks (CNNs) have been extensively studied for Hyperspectral Image Classification (HSIC). However, CNNs are critically attributed to a large number of labeled training samples, which outlays high costs in terms of time and resources. Moreover, CNNs are trained on some samples and have been tested on the entire HSI. Perhaps, the entire HSI is taken into account at test time to appropriately generate the ground truth maps. In order to obtain a higher accuracy while considering the limited availability of training samples and disjoint validation and test samples, this work proposes a fast and compact 3D CNN-based Active Learning (AL) for HSIC that integrates both deep transfer learning and AL into a unified framework. In the proposed methodology, a 3D CNN model is trained with very few training samples (i.e., 5%, only) and in the next phase, the most informative and heterogeneous samples are queried from the validation set (candidate set) based on the fuzziness, mutual information and breaking ties of the trained model. The 3D CNN model is later fine-tuned (rather retraining from scratch) with the new training samples (i.e., 200 samples are selected in each iteration) to reduce the computational cost. The proposed method has been compared with the state-of-the-art traditional and deep models proposed for HSIC. Experimental results proved the superiority of our proposed method on several benchmark HSI datasets with significantly fewer labeled samples. Matlab demo can be accessed on GitHub: github.com/mahmad00.

Currently, the different deep neural network (DNN) learning approaches have done much for the classification of hyperspectral images (HSIs), especially most of them use the convolutional neural network (CNN). HSI data have the characteristics of multidimensionality, correlation, nonlinearity, and a large amount of data. Therefore, it is particularly important to extract deeper features in HSIs by reducing dimensionalities which help improve the classification in both spectral and spatial domains. In this article, we present a spatial–spectral HSI classification algorithm, local similarity projection Gabor filtering (LSPGF), which uses local similarity projection (LSP)-based reduced dimensional CNN with a 2-D Gabor filtering algorithm. First, use the local similarity analysis to reduce the dimensionality of the hyperspectral data, and then we use the 2-D Gabor filter to filter the reduced hyperspectral data to generate spatial tunnel information. Second, use the CNN to extract features from the original hyperspectral data to generate spectral tunnel information. Third, the spatial tunnel information and the spectral tunnel information are fused to form the spatial–spectral feature information, which is input into the deep CNN to extract more effective features; and finally, a dual optimization classifier is used to classify the final extracted features. This article compares the performance of the proposed method with other algorithms in three public HSI databases and shows that the overall accuracy of the classification of LSPGF outperforms all datasets.

Model-based approaches to pansharpening still constitute a class of widely employed methods, thanks to their straightforward applicability to many problems, dispensing the user from time-consuming training phases. The injection scheme based on an accurate estimation (exploiting regression) of the relationship between the details contained in the PAN image and those required for the enhancement of the MS image represents the most updated approach to this problem, being characterized by both theoretical and practical optimality. We elaborated on this scheme by designing a procedure for estimating the key parameters required for the optimal setting of such regression-based approach. We tested this approach on several datasets acquired by the WorldView satellites comparing the proposed approach with a benchmark consisting of some state-of-the-art pansharpening methods.

Although the performance of pansharpening has been significantly improved by advanced deep-learning (DL) technologies in recent years, most DL-based methods fail to process multispectral (MS) images with arbitrary numbers of bands by a single model. Consequently, it is inevitable to train separate models for MS images with different numbers of bands, which is time- and storage-consuming as well as inefficient in practice. To tackle the above problem, we propose a bidirectional recurrent pansharpening network (named ArbRPN) for MS images with arbitrary numbers of bands. Our ArbRPN can dynamically reconstruct high-resolution (HR) MS images with different numbers of bands by adaptively changing the number of recurrence to the number of bands of the low-resolution (LR) MS images. Leveraging on the ability of the ArbRPN to process MS images with any number of bands, one can even customize the bands to be pansharpened. Moreover, to achieve superior performance, spectral discrepancy and dependence are considered in the ArbRPN. Details from the panchromatic (PAN) image are adaptively injected into the fused product according to the captured spectral dependence. Furthermore, training strategies of existing DL-based pansharpening methods can only group MS images with a constant number of bands into mini-batches. Therefore, we present a mask-based training method (called mask-training) to solve this problem. Benefiting from the mask-training, our ArbRPN can achieve superior performance and robustness during pansharpening. Extensive experiments show the superior performance of our ArbRPN with respect to the state-of-the-art (SOTA) methods applied to MS images with different numbers of bands. The code of our ArbRPN is available onhttps://github.com/Lihui-Chen/ArbRPN.git.

In recent years, deep learning (DL) has attracted increasing attention in hyperspectral unmixing (HU) applications due to its powerful learning and data fitting ability. The autoencoder (AE) framework, as an unmixing baseline network, achieves good performance in HU by automatically learning low-dimensional embeddings and reconstructing data. Nevertheless, the conventional AE-based architecture, which focuses more on the pixel-level reconstruction loss, tends to lose some significant detailed information of certain materials (e.g., material-related properties) in the reconstruction process. Therefore, inspired by the perception mechanism, we propose a cycle-consistency unmixing network, called CyCU-Net, by learning two cascaded AEs in an end-to-end fashion, to enhance the unmixing performance more effectively. CyCU-Net is capable of reducing the detailed and material-related information loss in the process of reconstruction by relaxing the original pixel-level reconstruction assumption to cycle consistency dominated by the cascaded AEs. More specifically, cycle consistency can be achieved by a newly proposed self-perception loss, which consists of two spectral reconstruction terms and one abundance reconstruction term. By taking advantage of the self-perception loss in the network, the high-level semantic information can be well preserved in the unmixing process. Moreover, we investigate the performance gain of CyCU-Net with extensive ablation studies. Experimental results on one synthetic and three real hyperspectral data sets demonstrate the effectiveness and competitiveness of the proposed CyCU-Net in comparison with several state-of-the-art unmixing algorithms.

Tensor-based methods have been widely studied to attack inverse problems in hyperspectral imaging since a hyperspectral image (HSI) cube can be naturally represented as a third-order tensor, which can perfectly retain the spatial information in the image. In this article, we extend the linear tensor method to the nonlinear tensor method and propose a nonlinear low-rank tensor unmixing algorithm to solve the generalized bilinear model (GBM). Specifically, the linear and nonlinear parts of the GBM can both be expressed as tensors. Furthermore, the low-rank structures of abundance maps and nonlinear interaction abundance maps are exploited by minimizing their nuclear norm, thus taking full advantage of the high spatial correlation in HSIs. Synthetic and real-data experiments show that the low rank of abundance maps and nonlinear interaction abundance maps exploited in our method can improve the performance of the nonlinear unmixing. A MATLAB demo of this work will be available athttps://github.com/LinaZhuangfor the sake of reproducibility.

Deep learning (DL) has aroused wide attention in hyperspectral unmixing (HU) owing to its powerful feature representation ability. As a representative of unsupervised DL approaches, autoencoder (AE) has been proven to be effective to better capture nonlinear components of hyperspectral images than the traditional model-driven linearized methods. However, only using hyperspectral images for unmixing fails to distinguish objects in complex scene, especially for different endmembers with similar materials. To overcome this limitation, we propose a novel multimodal unmixing network for hyperspectral images, called MUNet, by considering the height differences of light detection and ranging (LiDAR) data in a squeeze-and-excitation (SE)-driven attention fashion to guide the unmixing process, yielding performance improvement. MUNet is capable of fusing multimodal information and using the attention map derived by LiDAR to aid network that focuses on more discriminative and meaningful spatial information regarding scenes. Moreover, attribute profile (AP) is adopted to extract the geometrical structures of different objects to better model the spatial information of LiDAR. Experimental results on synthetic and real datasets demonstrate the effectiveness and superiority of the proposed method compared with several state-of-the-art unmixing algorithms. The codes will be available athttps://github.com/hanzhu97702/IEEE_TGRS_MUNet, contributing to the remote sensing community.

Standard hyperspectral (HS) pansharpening rely on fusion to enhance low-resolution HS (LRHS) images to the resolution of their matching panchromatic (PAN) images, whose practical implementation is normally under a stipulation of scale invariance of model across the training phase and the pansharpening phase. By contrast, arbitrary resolution HS (ARHS) pansharpening seeks to pansharpen LRHS images to any user-customized resolutions. For such a new HS pansharpening task, it is not feasible to train and store CNN models for all possible candidate scales, which implies the single model acquired from the training phase should be capable of being generalized to yield HS images with any resolutions in the pansharpening phase. To address the challenge, a novel variable sub-pixel convolution (VSPC)-based CNN (VSPC-CNN) method following our arbitrary upsampling CNN (AU-CNN) framework is developed for ARHS pansharpening. The VSPC-CNN method comprises a two-stage elevating thread. The first stage is to improve the spatial resolution of input HS image to that of the PAN image through a pre-pansharpening module and then a VSPC-encapsulated arbitrary scale attention upsampling (ASAU) module is cascaded for arbitrary resolution adjustment. After training with given scales, it can be generalized to pansharpen HS image to arbitrary scales under the spatial patterns invariance across the training and pansharpening phases. Experimental results from several specific VSPC-CNNs on both simulated and real HS datasets show the superiority of the proposed method.

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Owing to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers, and propose a novel backbone network called \ul{SpectralFormer}. Beyond band-wise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding group-wise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layer-wise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse "soft" residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixel- and patch-wise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

Sparse subspace clustering (SSC) is a representative data clustering paradigm that has been broadly applied in the unsupervised classification of hyperspectral images (HSIs). Existing SSC methods usually produce a subspace affinity matrix between representations of hyperspectral pixels first, followed by spectral clustering for the affinity matrix. To this end, the separated framework fails to exploit the dualities contained in both features and pixels or higher order entities at the same time, and thus, it is difficult to compute coclusters simultaneously. In addition, SSC methods often require expensive computational consumption and memory capacity to approximate the spectral decomposition of the affinity matrix, thus hindering the applicability of SSC methods for large HSIs. To overcome these limitations, we propose a novel graph convolutional sparse subspace coclustering (GCSSC) model with nonnegative orthogonal factorization for large HSIs in which affinity matrix learning and spectral coclustering are integrated into a unified optimizing model to obtain the optimal clustering results. Specifically, to form a more compact self-representation, the superpixel-based adaptive dictionary construction strategy is proposed instead of the global dictionary to precisely represent the pixels. To explore the spatial–contextual and spectral neighboring characteristics between dictionary atoms, graph convolution is incorporated into the dictionary atoms to aggregate the local neighborhood information, and the affinity matrix in the proposed coclustering framework is constructed under a joint sparsity constrained representation model. To reduce high computational consumption and memory capacity, a nonnegative orthogonal factorization constraint is proposed to offer an alternative spectral clustering for hyperspectral pixels and dictionary atoms simultaneously. The clustering performance of the proposed method is evaluated for three classical HSIs, and the experimental results illustrate that the proposed method is memory and computationally efficient and outperforms the state-of-the-art HSI clustering methods.

Clustering large hyperspectral images (HSIs) is a very challenging problem because large HSIs have high dimensionality, large spectral variability, and large computational and memory consumption. Recently, sparse subspace clustering (SSC) has achieved remarkable success in HSI clustering. However, most SSC-based methods suffer from the following bottlenecks for large HSIs: 1) high computational consumption and memory space during the construction of the similarity matrix and decomposition of the graph Laplacian matrix and 2) failure to capture the relationships among dictionary atoms, sparse coefficients, and hyperspectral pixels. To address these challenges, we propose a novel algorithm that extends SSC to cocluster large HSIs, called bipartite graph partition with graph nonnegative matrix factorization (BGP-GNMF). Specifically, to fully explore the characteristics of the spectral and spatial contexts in HSIs, we propose a novel superpixel and pixel coclustering framework with bipartite graph partitioning in the joint sparse representation domain, where superpixel-based dictionary atoms are defined as disjoint vertex sets of the bipartite graph and the joint sparsity representation is mapped into the adjacency matrix of the undirected bipartite graph. To overcome the challenges of high computational consumption and large memory space for large HSIs, the bipartite graph partition with orthonormal constrained nonnegative matrix factorization is proposed to simultaneously cluster the structured dictionary atoms and hyperspectral pixels with an indicator matrix. Finally, to exploit the intrinsic geometry of HSIs, we incorporate manifold regularization into the bipartite graph partition to improve final clustering accuracy. The effectiveness and efficiency of the proposed method are verified on three classical HSIs, and the experimental results illustrate the superiority of the proposed method compared with other state-of-the-art HSI clustering methods.

Hyperspectral anomaly detection aims to detect objects significantly different from their surrounding background. Recently, many detectors based on autoencoder (AE) exhibited promising performances in hyperspectral anomaly detection tasks. However, the fundamental hypothesis of the AE-based detector that anomaly is more challenging to be reconstructed than background may not always be true in practice. We demonstrate that an autoencoder could well reconstruct anomalies even without anomalies for training. Because AE models mainly focus on the quality of sample reconstruction and do not care if the encoded features solely represent the background rather than anomalies. If more information is preserved than needed to reconstruct the background, the anomalies will be well reconstructed. This paper proposes a cluster-memory augmented autoencoder via deep optimal transportation clustering (OTCMA) for hyperspectral anomaly detection to solve this problem. The deep clustering method based on optimal transportation is proposed to enhance the features consistency of samples within the same categories and features discrimination of samples in different categories. The memory module stores the background’s consistent features, which are the cluster centers for each category background. We retrieve more consistent features from the memory module instead of reconstructing a sample utilizing its own encoded features. The network focuses more on consistent feature reconstruction by training AE with a memory module. This effectively restricts the reconstruction ability of AE and prevents reconstructing anomalies. Extensive experiments on the benchmark datasets demonstrate that our proposed OTCMA achieves state-of-the-art results. Besides, this paper presents further discussions about the effectiveness of our proposed memory module and different criterion for better anomaly detection.

Recently, deep learning-based methodologies have attained unprecedented performance in hyperspectral (HS) pansharpening, which aims to improve the spatial quality of HS images (HSIs) by making use of details extracted from the high-resolution panchromatic (HR-PAN) image. However, it remains challenging to incorporate the details into the pansharpened image effectively, while alleviating the spectral distortion simultaneously. To tackle this problem, in this article, we propose an adaptive feature modulation-based detail injection network (AFM-DIN) for HS pansharpening, which mainly consists of four phases: high-frequency details generation of the HR-PAN image, multiscale feature extraction of the upsampled HSI, AFM-based detail injection and reconstruction of the HR-HSI. First, a novel octave convolution unit is employed to decompose the HR-PAN image into high and low frequencies, and then merge the high-frequency features together to generate the comprehensive PAN-details. Second, the spatial and spectral separable 3D convolution units with multiple kernel sizes are designed to extract multiscale features of the upsampled HSI in a computationally efficient manner. Subsequently, by taking the critical PAN-details as prior, the proposed AFM module is able to not only incorporate the detail information effectively, but also adjust the injected details adaptively to ensure the spectral fidelity. Finally, the anticipated HR-HSI is obtained through adding the upsampled HSI to the predicted HSI-details reconstructed from informative modulated features. Extensive comparison experiments with several state-of-the-arts conducted on simulated and real HS data sets demonstrate that our proposed AFM-DIN can achieve superior pansharpening accuracy in both spatial and spectral aspects.

Large-scene precise classification of multispectral images (MSIs) has become one of the hot topics in remote sensing field. MSIs usually have wide swath and a meter or even submeter level of spatial resolution, which make large-scene observation possible. However, the limited number of spectral bands leads to the confusion of land covers in classification, especially for the large-scene conditions with abundant land cover types. Therefore, overlapped hyperspectral images (HSIs) can be used to improve the precision degree of classification. To achieve this purpose, coupled dictionary learning has been proposed as a major means. Aiming at separating the class-specific characteristics and mutual patterns among different land covers, this paper proposed a separable coupled dictionary learning (SCDL) method, which converts the separation of mutual features into the construction of separable coupled dictionaries and learns both class-specific coupled dictionaries and mutual coupled dictionaries simultaneously with the aid of label information. More specifically, the proposed method uses the labels of training samples to construct class-specific reconstruction error constraint, class-specificity constraint and separable dictionary incoherence constraint as regularization terms, to make sure that the learned coupled dictionaries to be both compact and discriminative. The learned separable coupled dictionaries facilitate pixels belong to the same category to be represented by the mutual dictionary and the class-specific sub-dictionary of corresponding class. The experiments compared with several state-of-the-art methods on three pairs of HSI and MSI have shown better classification performance.

Modality translation, which aims to translate images from a source modality to a target one, has attracted a growing interest in the field of remote sensing recently. Compared to translation problems in multimedia applications, modality translation in remote sensing often suffers from inherent ambiguities, i.e., a single input image could correspond to multiple possible outputs, and the results may not be valid in the following image interpretation tasks, such as classification and change detection. To address these issues, we make the attempt to utilizing time-series data to resolve the ambiguities. We propose a novel multimodality image translation framework, which exploits temporal information from two aspects: 1) by introducing a guidance image from given temporally neighboring images in the target modality, we employ a feature mask module and transfer semantic information from temporal images to the output without requiring the use of any semantic labels and 2) while incorporating multiple pairs of images in time series, a temporal constraint is formulated during the learning process in order to guarantee the uniqueness of the prediction result. We also build a multimodal and multitemporal dataset that contains synthetic aperture radar (SAR), visible, and short-wave length infrared band (SWIR) image time series of the same scene to encourage and promote research on modality translation in remote sensing. Experiments are conducted on the dataset for two cross-modality translation tasks (SAR to visible and visible to SWIR). Both qualitative and quantitative results demonstrate the effectiveness and superiority of the proposed model.

Low-rank-tensor-approximation (LRTA)-based hyperspectral image (HSI) restoration has drawn increasing attention. However, most of the methods construct a hidden low-rank tensor by utilizing the non-local self-similarity (NLSS) and global spectral correlation (GSC) inherited by HSIs. Although achieving state-of-the-art (SOTA) restoration performance, NLSS and GSC have limitations. NLSS is introduced from natural image denoising to remove spatially independent identically distributed (i.i.d.) Gaussian and impulse noise. While GSC, which is naturally possessed by HSIs, is adopted to maintain the spectral integrity and remove spectrally, i.i.d., degradations. Therefore, NLSS and GSC may not be successfully used for complex HSI restoration tasks, such as destriping, cloud removal and recovery of atmospheric absorption bands. To solve the issue, borrowing the idea from manifold learning, the geometry information characterized by proximity relationship, is integrated with the LRTA to solve the above issue, named as multi-graph-based LRTA (MGLRTA). Different with most of the existing methods, the proposed MGLRTA directly models an HSI as a low-rank tensor and efficiently explores the extra proximity information on the defined graphs that are not only inherited by the low-rank constraints but also naturally possessed in HSIs. A well-posed iterative algorithm is designed to solve the restoration problem. Experimental results on different datasets that cover several severe degradation scenarios demonstrate that the proposed MGLRTA outperforms the SOTA HSI restoration methods.

A hyperspectral image sequence can be obtained at different time in the same region from a hyperspectral sensor. The environmental change usually leads to variation in endmember reflectance, which has an important influence on unmixing process. In this article, a Bayesian unmixing model considering spectral variability for hyperspectral sequence is proposed, in which composite prior distributions of abundance and endmember variability are developed. The abundance priors consider the continuity of abundance in the temporal and spatial domains, simultaneously. Specifically, in the spatial domain, a data-adaptive variance of the abundance prior distribution is put forward based on local spatial difference. Moreover, the priors of endmember variability in temporal continuity and spectral smoothness are also exploited. Finally, a joint posterior distribution is obtained by the likelihood function and the parameter prior distributions, which can be calculated by the Markov chain Monte Carlo (MCMC) algorithm. Experiments on synthetic and real data sets demonstrate the effectiveness of the proposed approach in terms of abundance, endmember, and its variability estimation accuracy.

An important task of spectral unmixing is to recover the signatures of endmembers from a hyperspectral dataset in which no pure signature is exposed. Most algorithms are based on linear mixing model and require that data should be sufficiently scattered to guarantee model uniqueness. However, data may not be scattered sufficiently enough and become incomplete. In this case, can we still recover the endmembers accurately? Moreover, if we wrongly estimate an endmember, the error may be propagated to other endmembers. For these purposes, we propose a new model, namely, affine simplicial cone (ASC), to capture the local geometric feature of data. This model requires less geometric information and relaxes the conditions of model uniqueness. Then, we present analyses on the error propagation of turbulent endmembers and the condition of local uniqueness. Based on ASC, we present an endmember purification problem to recover only one endmember. In this way, error propagation can be alleviated, and local uniqueness can be easily satisfied. Finally, we develop an endmember purifying algorithm (EPA) to solve this problem. Our experiments demonstrate that the performance of EPA is competitive to the state-of-the-art unmixing algorithms not only for the synthetic datasets but also for the real hyperspectral remote sensing datasets. We can conclude that the ASC model and the EPA algorithm have the potential capability for hyperspectral data exploration.

Hyperspectral images (HSIs) provide rich spectral information that has contributed to the successful performance improvement of numerous computer vision and remote sensing tasks. However, it can only be achieved at the expense of images’ spatial resolution. HSI super-resolution (HSI-SR), thus, addresses this problem by fusing low-resolution (LR) HSI with the multispectral image (MSI) carrying much higher spatial resolution (HR). Existing HSI-SR approaches require the LR HSI and HR MSI to be well registered, and the reconstruction accuracy of the HR HSI relies heavily on the registration accuracy of different modalities. In this article, we propose an unregistered and unsupervised mutual Dirichlet-Net ($u^{2}$-MDN) to exploit the uncharted problem domain of HSI-SRwithout the requirement of multimodality registration. The success of this endeavor would largely facilitate the deployment of HSI-SR since registration requirement is difficult to satisfy in real-world sensing devices. The novelty of this work is threefold. First, to stabilize the fusion procedure of two unregistered modalities, the network is designed to extract spatial information and spectral information of two modalities with different dimensions through a shared encoder–decoder structure. Second, the mutual information (MI) is further adopted to capture the nonlinear statistical dependencies between the representations from two modalities (carrying spatial information) and their raw inputs. By maximizing the MI, spatial correlations between different modalities can be well characterized to further reduce the spectral distortion. We assume that the representations follow a similar Dirichlet distribution for their inherent sum-to-one and nonnegative properties. Third, a collaborative$l_{2,1}$-norm is employed as the reconstruction error instead of the more common$l_{2}$-norm to better preserve the spectral information. Extensive experimental results demonstrate the superior performance of$u^{2}$-MDN as compared to the state of the art.

Hyperspectral target detection can be described as locating targets of interest within a hyperspectral image based on prior information of targets. The complexity of actual scenes limits the performance of traditional statistical methods that rely on model assumptions, and traditional machine learning methods rely on mapping functions with limited complexity. To address these problems, we propose a Siamese transformer network for hyperspectral image target detection (STTD). The contribution of this article is threefold. First, we propose a novel method of constructing training samples using only the image itself and the limited prior information, which is suitable for target detection based on the Siamese network framework. Second, the Siamese network framework is utilized to solve the problem of similarity metric learning, i.e., make homogeneous features as close as possible and heterogeneous features as far as possible. Third, the most state-of-the-art network, transformer, is applied as the backbone of our proposed Siamese network to extract global features from spectra with long-range dependencies to achieve target detection. Furthermore, we make adaptive improvements to transformer for hyperspectral images. The proposed method shows its unique advantages in suppressing the background to a low level and highlighting the target with high probability. Experiments on five different datasets demonstrate the superiority of the proposed STTD as compared to the state-of-the-art.

In this article, we propose a minimum simplex convolutional network (MiSiCNet) for deep hyperspectral unmixing. Unlike all the deep learning-based unmixing methods proposed in the literature, the proposed convolutional encoder–decoder architecture incorporates spatial information and geometrical information of the hyperspectral data in addition to the spectral information. The spatial information is incorporated using convolutional filters and implicitly applying a prior on the abundances. The geometrical information is exploited by incorporating a minimum simplex volume penalty term in the loss function for the endmember estimation. This term is beneficial when there are no pure material pixels in the data, which is often the case in real-world applications. We generated simulated datasets, where we consider two different no-pure pixel scenarios. In the first scenario, there are no pure pixels but at least two pixels on each facet of the data simplex (i.e., mixtures of two pure materials). The second scenario is a complex case with no pure pixels and only one pixel on each facet of the data simplex. In addition, we evaluate the performance of MiSiCNet in three real datasets. The experimental results confirm the robustness of the proposed method to both noise and the absence of pure pixels. In addition, MiSiCNet considerably outperforms the state-of-the-art unmixing approaches. The results are given in terms of spectral angle distance in degree for the endmember estimation and the root mean square error in percentage for the abundance estimation. MiSiCNet was implemented in Python (3.8) using PyTorch as the platform for the deep network and is available online:https://github.com/BehnoodRasti/MiSiCNet.

Convolutional Neural Networks (CNNs) have been extensively utilized for Hyperspectral (HSI) as well as Light Detection and Ranging (LiDAR) data Classification. However, CNNs have not been much explored for joint HSI and LiDAR image classification. Therefore, this article proposes a joint feature learning (HSI and LiDAR) and fusion mechanism using CNN and Spatial Morphological blocks which generates highly accurate land-cover maps. The CNN model comprises three Conv3D layers and is directly applied to the HSIs for extracting discriminative spectral-spatial feature representation. On the contrary, the spatial morphological block is able to capture the information relevant to the height or shape of the different land-cover regions from LiDAR data. The LiDAR features are extracted using morphological dilation and erosion layers which increase the robustness of the proposed model by considering elevation information as an additional feature. Finally, both the obtained features from CNNs and spatial morphological blocks are combined using an additive operation prior to the classification. Extensive experiments are shown with widely used HSIs and LiDAR datasets, i.e., University of Houston (UH), Trento, and MUUFL Gulfport scene. The reported results show that the proposed model significantly outperforms traditional methods and other state-of-the-art deep learning models. The source code for the proposed model will be made available publicly at https://github.com/AnkurDeria/HSI+LiDAR.

The hyperspectral images are composed of a variety of textures across the different bands which increase the spectral similarity and make it difficult to predict the pixel-wise labels without inducing additional complexity at the feature level. To extract robust and discriminative features from the different regions of land cover, the hyperspectral research community is still seeking such type of convolutions which can efficiently deal with fine-grained texture information during the feature extraction phase, which often overlook this aspect by vanilla convolution. To overcome the above shortcoming, this article proposes a generalized gradient centralized 3D convolution (G2C-Conv3D) operation, which is a weighted combination between the vanilla and gradient centralized 3D convolutions (GC-Conv3D) to extract both theintensity-levelsemantic information andgradient-levelinformation. This can be easily plugged into the existing HSI feature extraction networks to boost the performance of accurate prediction for land-cover types. To validate the feasibility of the proposedG2C-Conv3D, we have considered the existing CNN3D, MS3DNet, ContextNet, and SSRN feature extraction models and as well as CAE3D, VAE3D, and SAE3D autoencoder (AE) networks, respectively. All these networks are embedded withG2C-Conv3Dconvolution to implement both generalized gradient centralized feature extraction networks (G2C-FE) and generalized gradient centralized AE networks (G2C-AE) for fine-grained spectral–spatial feature learning. In addition,G2C-Conv2Dis also considered with few networks. The extensive experiments are conducted on four most widely used hyperspectral datasets i.e., IP, KSC, UH, and UP, respectively, and compared with the nine methods. The results demonstrate that the proposedG2C-Conv3Dcan effectively enhance the feature learning ability of the existing networks and both the qualitative and quantitative results show the superiority and effectiveness of the proposedG2C-Conv3D. The source codes will be publicly available athttps://github.com/danfenghong/G2C-Conv3D-HSI.

Object detection is a challenging task in remote sensing because objects only occupy a few pixels in the images, and the models are required to simultaneously learn object locations and detection. Even though the established approaches well perform for the objects of regular sizes, they achieve weak performance when analyzing small ones or getting stuck in the local minima (e.g. false object parts). Two possible issues stand in their way. First, the existing methods struggle to perform stably on the detection of small objects because of the complicated background. Second, most of the standard methods used hand-crafted features, and do not work well on the detection of objects parts of which are missing. We here address the above issues and propose a new architecture with a multiple patch feature pyramid network (MPFP-Net). Different from the current models that during training only pursue the most discriminative patches, in MPFPNet the patches are divided into class-affiliated subsets, in which the patches are related and based on the primary loss function, a sequence of smooth loss functions are determined for the subsets to improve the model for collecting small object parts. To enhance the feature representation for patch selection, we introduce an effective method to regularize the residual values and make the fusion transition layers strictly norm-preserving. The network contains bottom-up and crosswise connections to fuse the features of different scales to achieve better accuracy, compared to several state-of-the-art object detection models. Also, the developed architecture is more efficient than the baselines.

Detection of objects is extremely important in various aerial vision-based applications. Over the last few years, the methods based on convolution neural networks (CNNs) have made substantial progress. However, because of the large variety of object scales, densities, and arbitrary orientations, the current detectors struggle with the extraction of semantically strong features for small-scale objects by a predefined convolution kernel. To address this problem, we propose the rotation equivariant feature image pyramid network (REFIPN), an image pyramid network based on rotation equivariance convolution. The proposed model adopts single-shot detector in parallel with a lightweight image pyramid module (LIPM) to extract representative features and generate regions of interest in an optimization approach. The proposed network extracts feature in a wide range of scales and orientations by using novel convolution filters. These features are used to generate vector fields and determine the weight and angle of the highest-scoring orientation for all spatial locations on an image. By this approach, the performance for small-sized object detection is enhanced without sacrificing the performance for large-sized object detection. The performance of the proposed model is validated on two commonly used aerial benchmarks and the results show our proposed model can achieve state-of-the-art performance with satisfactory efficiency.

Graph-based methods are promising approaches for traditional and modern techniques in change detection (CD) applications. Nonetheless, some graph-based approaches omit the existence of useful priors that account for the structure of a scene, and the inter- and intra-relationships between the pixels are analyzed. To address this issue, in this article, we propose a framework for CD based on graph fusion and driven by graph signal smoothness representation. In addition to modifying the graph learning stage, in the proposed model, we apply a Gaussian mixture model for superpixel segmentation (GMMSP) as a downsampling module to reduce the computational cost required to learn the graph of the entire images. We carry out tests on 14 real cases of natural disasters, farming, and construction. The dataset contains homogeneous cases with multispectral (MS) and synthetic aperture radar (SAR) images, along with heterogeneous cases that include MS/SAR images. We compare our approach against probabilistic thresholding, unsupervised learning, deep learning, and graph-based methods. In terms of Cohen’s kappa coefficient, our proposed model based on graph signal smoothness representation outperformed state-of-the-art approaches in ten out of 14 datasets.

In recent years, enormous research has been made to improve the classification performance of single-modal remote sensing (RS) data. However, with the ever-growing availability of RS data acquired from satellite or airborne platforms, simultaneous processing and analysis of multimodal RS data pose a new challenge to researchers in the RS community. To this end, we propose a deep-learning-based new framework for multimodal RS data classification, where convolutional neural networks (CNNs) are taken as a backbone with an advanced cross-channel reconstruction module, called CCR-Net. As the name suggests, CCR-Net learns more compact fusion representations of different RS data sources by the means of the reconstruction strategy across modalities that can mutually exchange information in a more effective way. Extensive experiments conducted on two multimodal RS datasets, including hyperspectral (HS) and light detection and ranging (LiDAR) data, i.e., the Houston2013 dataset, and HS and synthetic aperture radar (SAR) data, i.e., the Berlin dataset, demonstrate the effectiveness and superiority of the proposed CCR-Net in comparison with several state-of-the-art multimodal RS data classification methods. The codes will be openly and freely available athttps://github.com/danfenghong/IEEE_TGRS_CCR-Netfor the sake of reproducibility.

The low resolution of remote sensing images often limits the land cover classification (LCC) performance. Super resolution (SR) can improve the image resolution, while greatly increasing the computational burden for the LCC due to the larger size of the input image. In this article, the SR-guided deep network (SRGDN) framework is proposed, which can generate meaningful structures from higher resolution images to improve the LCC performance without consuming more computational costs. In general, the SRGDN consists of two branches (i.e., SR branch and LCC branch) and a guidance module. The SR branch aims to increase the resolution of remote sensing images. Since high- and low-resolution image pairs cannot be directly provided by imaging sensors to train the SR branch, we introduce a self-supervised generative adversarial network (GAN) to estimate the downsampling kernel that can produce these image pairs. The LCC branch adopts the high-resolution network (HRNet) to retain as much resolution information with a few downsampling operations as possible. The guidance module teaches the LCC branch to learn the high-resolution information from the SR branch without the utilization of the higher-resolution images as the inputs. Furthermore, the guidance module introduces spatial pyramid pooling (SPP) to match the feature maps of different sizes in the two branches. In the testing stage, the guidance module and SR branch can be removed, and therefore do not create additional computational costs. Experimental results on three real datasets demonstrate the superiority of the proposed method over several well-known LCC approaches.

Deep learning works normally in PolSAR image classification because the complex terrain scattering characteristic results in large intraclass differences and high interclass similarity. Deep metric learning (DML) aims to make the features keep a closer intraclass and a farther interclass distance. Therefore, we introduce DML and then propose an N-cluster generative adversarial net (N-cluster GAN) framework for PolSAR image classification. However, existing DML losses mainly focus on the relationship between individual samples in feature space. Hence, we propose N-cluster loss that pays more attention to the overall structure of all samples. Meanwhile, traditional hard negative sample mining methods occupy lots of computational resources. In addition, the hard level of the negative samples will affect the model’s performance. Therefore, we explore a new method based on a GAN framework to replace the sample mining. Positive N-cluster loss is added to the discriminator ($D$), and a negative one is added to the generator ($G$). In this way,$D$will possess better classification ability, and$G$can produce hard negative samples for$D$. Then, the hard level of the generated negative samples will change with the discrimination of$D$, which is appropriate for the proposed model. N-cluster loss can be directly calculated through the extracted features rather than redundant data preparation. The proposed model is verified on four PolSAR datasets from two aspects of the loss function and negative samples mining. Then, it achieves competitive performance compared with state-of-the-art algorithms.

Target decomposition features are the cornerstone of subsequent analyses for PolSAR images. Generally, adopting single or several decomposition algorithms limits the representation ability for original terrain characteristics. Using all the existing decomposition features, however, will definitely increase computational complexity. Besides, some features even have a negative effect on the following tasks. To address these problems, a sparse variational autoencoder feature selection framework (SVAE-FS) is proposed in this article. In detail, the encoder transforms the original feature set into latent space and then decoder reconstructs the corresponding pseudo set on this latent space. Similarly, a pseudo subset is subsequently obtained by the SVAE. The discrepancy, namely reconstruction error, between the pseudo set and the pseudo subset is taken as an evaluation criterion which reflects the feature representation ability of pseudo subset. Sparse constraint in the encoder makes the representative features stand out. Meanwhile, the linear feature transformation layer of the encoder enables the SVAE to evaluate different scale subsets without repeated training. Finally, a greedy selection approach with search scale$K$is proposed to find the suboptimal subset. This procedure not only reduces time consumption, but also ensures the performance of the subset. The selected features are analyzed on four real PolSAR datasets according to the terrain scattering characteristics. Furthermore, these features have achieved competitive performance on three PolSAR image tasks.

Model-based polarimetric target decomposition (TD) generally solves scattering components and parameters under pre-set decomposition base, then decomposition features are also obtained. However, pre-set base could not be adjusted according to different scenes. Furthermore, solving the polarimetric parameters needs to explore additional information or consider limiting conditions to build equations, which is hard and easily to bring negative effects into decomposition features. To this end, we regard the TD as a process of learning decomposition base and features by deep learning. Then, the polarimetric decomposition feature learning (PDFL) model is proposed in this paper. Strictly, this model is not an incoherent TD method but a learning-based method. It dose not need to construct the parameter solution equations or fixed base. Then, the decomposition base and feature can be adaptively learned according to scattering characteristics of current dataset. Due to the characteristics of unsupervised reconstruction, deep auto encoder (DAE) is used as the model foundation. Then, some adjustments and constraints are utilized to make the DAE fit closely with TD. The encoder extracts latent vector from PolSAR data, then the decoder reconstructs pseudo data on this latent vector. The reconstruction can be regarded as the inverse process of TD, so the base matrix of decoder and the latent vector indicate the learned decomposition base and features when the model converges. The effectiveness of PDFL is verified on simulated and real PolSAR datasets. Compared with representative algorithms, proposed model gains more discriminative features and achieves competitive performance on terrain classification and segmentation tasks.

Owing to the powerful data representation ability of deep learning (DL) techniques, tremendous progress has been recently made in hyperspectral image (HSI) classification. Convolutional neural network (CNN), as a main part of the DL family, has been proven to be considerably effective to extract spatial-spectral features for HSIs. Nevertheless, its classification performance, to a great extent, depends on the quality and quantity of samples in the network training process. To select those samples, either labeled or unlabeled, that can be used to enhance the generalization ability of CNNs and further improve the classification accuracy, we propose an iterative semi-supervised CNNs framework by means of active learning and superpixel segmentation techniques, dubbed as semi-active CNNs (SA-CNNs), for HSI classification. More specifically, we start to pre-train a CNNs-based model on a small-scale unbiased labeled set and infer unlabeled data using the trained model, i.e., generating pseudo-labels. Then, the reliable samples, which consist of two parts: high label-homogeneity and most informativeness, are actively selected from superpixel segments. These selected labeled and unlabeled samples with their labels and pseudo-labels are re-fed into the next-round network training. Moreover, three different schedules, i.e.,log-,exp-, andlinear-schedules, are progressively adopted to fully explore their potentials in sample selection, until a labeling budget is finally reached. Extensive experiments are conducted on three benchmark HSI datasets, demonstrating substantial performance improvements of the proposed SA-CNNs over other similar competitors.

Blind hyperspectral unmixing (HU) has long been recognized as a crucial component in analyzing the hyperspectral imagery (HSI) collected by airborne and spaceborne sensors. Due to the highly ill-posed problems of such a blind source separation scheme and the effects of spectral variability in hyperspectral imaging, the ability to accurately and effectively unmixing the complex HSI still remains limited. To this end, this article presents a novel blind HU model, called sparsity-enhanced convolutional decomposition (SeCoDe), by jointly capturing spatial–spectral information of HSI in a tensor-based fashion. SeCoDe benefits from two perspectives. On the one hand, the convolutional operation is employed in SeCoDe to locally model the spatial relation between the targeted pixel and its neighbors, which can be well explained by spectral bundles that are capable of addressing spectral variabilities effectively. It maintains, on the other hand, physically continuous spectral components by decomposing the HSI along with the spectral domain. With sparsity-enhanced regularization, an alternative optimization strategy with alternating direction method of multipliers (ADMM)-based optimization algorithm is devised for efficient model inference. Extensive experiments conducted on three different data sets demonstrate the superiority of the proposed SeCoDe compared to previous state-of-the-art methods. We will also release the code athttps://github.com/danfenghong/IEEE_TGRS_SeCoDeto encourage the reproduction of the given results.

This article proposes a denoising method based on sparse spectral–spatial and low-rank representations (SSSLRR) using the 3-D orthogonal transform (3-DOT). SSSLRR can be effectively used to remove the Gaussian and mixed noise. SSSLRR uses 3-DOT to decompose noisy HSI to sparse transform coefficients. The 3-D discrete orthogonal wavelet transform (3-D DWT) is a representative 3-DOT suitable for denoising since it concentrates on the signal in few transform coefficients, and the 3-D discrete orthogonal cosine transform (3-D DCT) is another example. An SSSLRR using 3-D DWT will be called SSSLRR-DWT. SSSLRR-DWT is an iterative algorithm based on the alternating direction method of multipliers (ADMM) that uses sparse and nuclear norm penalties. We use an ablation study to show the effectiveness of the penalties we employ in the method. Both simulated and real hyperspectral datasets demonstrate that SSSLRR outperforms other comparative methods in quantitative and visual assessments to remove the Gaussian and mixed noise.

Due to the limitations of imaging systems, satellite hyperspectral imagery (HSI), which yields rich spectral information in many channels, often suffers from poor spatial resolution. HSI super-resolution (SR) refers to the fusion of high spatial resolution multispectral imagery (MSI) and low spatial resolution HSI to generate HSI that has both a high spatial and high spectral resolution. However, most existing SR methods assume that the two original images used are perfectly registered: in reality, nonrigid deformation areas can exist locally in the two images even if prior registration of the control points has been carried out. To address this problem, we propose a novel unsupervised spectral unmixing and image deformation correction network—NonRegSRNet—with multimodal and multitask learning that can be used for the joint registration of HSI and MSI and to produce SR imagery. More specifically, NonRegSRNet integrates the dense registration and SR tasks into a unified model that includes a triplet convolutional neural network. This allows these two tasks to complement each other so that better registration and SR results can be achieved. Furthermore, because the point spread function (PSF) and spectral response function (SRF) are often unavailable, two special convolutional layers are designed to adaptively learn the parameters of the PSF and SRF, which makes the proposed model more adaptable. Experimental results demonstrate that the proposed method has the ability to produce highly accurate and stable reconstructed images under complex nonrigid deformation conditions. (Code available athttps://github.com/saber-zero/NonRegSRNet)

Despite recent advances achieved by deep learning techniques in the fusion of low-spatial-resolution hyperspectral image (LR-HSI) and high-spatial-resolution multispectral image (HR-MSI), it remains a challenge to reconstruct the high-spatial-resolution HSI (HR-HSI) with more accurate spatial details and less spectral distortions, since the low-level structure information such as sharp edges tends to be weakened or lost as the network depth grows. To tackle this issue, we creatively propose an edge-conditioned feature transform network (EC-FTN) in this article, which is mainly composed of three parts, namely, feature extraction network (FEN), feature fusion and transformation network (FFTN), and image reconstruction network (IRN). First, two computationally efficient FENs with 3-D convolutions and reshaping layers are employed to extract the joint spectral-spatial features of input images. Then, the FFTN conditioned on the edge map prior can fuse and transform the features adaptively, in which a fusion node and several cascaded feature modulation modules (FMMs) equipped with feature-wise modulation layers are constructed. Specifically, the edge map is generated via transfer learning, i.e., by applying the Sobel operator to feature maps of the red-green-blue (RGB) version of HR-MSI resulting from the pretrained VGG16 model without extra training. Finally, the desired HR-HSI is recovered from the transformed features through IRN. Furthermore, we elaborately design a weighted combinatorial loss function consisting of mean absolute error, image gradient difference, and spectral angle terms to guide the training. Experiments on both ground-based and remotely sensed datasets demonstrate that our EC-FTN outperforms state-of-the-art methods in visual and quantitive evaluations, as well as in fine details reconstruction.

Outlier detection is to separate anomalous data from inliers in the dataset. Recently, the most deep learning methods of outlier detection leverage an auxiliary reconstruction task by assuming that outliers are more difficult to recover than normal samples (inliers). However, it is not always true in deep auto-encoder (AE) based models. The auto-encoder based detectors may recover certain outliers even if outliers are not in the training data, because they do not constrain the feature learning. Instead, we think outlier detection can be done in the feature space by measuring the distance between outliers' features and the consistency feature of inliers. To achieve this, we propose an unsupervised outlier detection method using a memory module and a contrastive learning module (MCOD). The memory module constrains the consistency of features, which merely represent the normal data. The contrastive learning module learns more discriminative features, which boosts the distinction between outliers and inliers. Extensive experiments on four benchmark datasets show that our proposed MCOD performs well and outperforms eleven state-of-the-art methods.

Recently, embedding and metric-based few-shot learning (FSL) has been introduced into hyperspectral image classification (HSIC) and achieved impressive progress. To further enhance the performance with few labeled samples, we in this paper propose a novel FSL framework for HSIC with a class-covariance metric (CMFSL). Overall, the CMFSL learns global class representations for each training episode by interactively using training samples from the base and novel classes, and a synthesis strategy is employed on the novel classes to avoid overfitting. During the meta-training and meta-testing, the class labels are determined directly using the Mahalanobis distance measurement rather than an extra classifier. Benefiting from the task-adapted class-covariance estimations, the CMFSL can construct more flexible decision boundaries than the commonly used Euclidean metric. Additionally, a lightweight cross-scale convolutional network (LXConvNet) consisting of 3D and 2D convolutions is designed to thoroughly exploit the spectral-spatial information in the high-frequency and low-frequency scales with low computational complexity. Furthermore, we devise a spectral-prior-based refinement module (SPRM) in the initial stage of feature extraction, which cannot only force the network to emphasize the most informative bands while suppressing the useless ones, but also alleviate the effects of the domain shift between the base and novel categories to learn a collaborative embedding mapping. Extensive experiment results on four benchmark data sets demonstrate that the proposed CMFSL can outperform the state-of-the-art methods with few-shot annotated samples.

Over the past decades, enormous efforts have been made to improve the performance of linear or nonlinear mixing models for hyperspectral unmixing (HU), yet their ability to simultaneously generalize various spectral variabilities (SVs) and extract physically meaningful endmembers still remains limited due to the poor ability in data fitting and reconstruction and the sensitivity to various SVs. Inspired by the powerful learning ability of deep learning (DL), we attempt to develop a general DL approach for HU, by fully considering the properties of endmembers extracted from the hyperspectral imagery, called endmember-guided unmixing network (EGU-Net). Beyond the alone autoencoder-like architecture, EGU-Net is a two-stream Siamese deep network, which learns an additional network from the pure or nearly pure endmembers to correct the weights of another unmixing network by sharing network parameters and adding spectrally meaningful constraints (e.g., nonnegativity and sum-to-one) toward a more accurate and interpretable unmixing solution. Furthermore, the resulting general framework is not only limited to pixelwise spectral unmixing but also applicable to spatial information modeling with convolutional operators for spatial-spectral unmixing. Experimental results conducted on three different datasets with the ground truth of abundance maps corresponding to each material demonstrate the effectiveness and superiority of the EGU-Net over state-of-the-art unmixing algorithms. The codes will be available from the website: https://github.com/danfenghong/IEEE_TNNLS_EGU-Net.

Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI's scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html.

Recently, the majority of successful matching approaches are based on convolutional neural networks, which focus on learning the invariant and discriminative features for individual image patches based on image content. However, the image patch matching task is essentially to predict the matching relationship of patch pairs, that is, matching (similar) or non-matching (dissimilar). Therefore, we consider that the feature relation (FR) learning is more important than individual feature learning for image patch matching problem. Motivated by this, we propose an element-wise FR learning network for image patch matching, which transforms the image patch matching task into an image relationship-based pattern classification problem and dramatically improves generalization performances on image matching. Meanwhile, the proposed element-wise learning methods encourage full interaction between feature information and can naturally learn FR. Moreover, we propose to aggregate FR from multilevels, which integrates the multiscale FR for more precise matching. Experimental results demonstrate that our proposal achieves superior performances on cross-spectral image patch matching and single spectral image patch matching, and good generalization on image patch retrieval.

In this paper we will review seminal contributions of Prof. Jose Bioucas Dias for the improvement of the spatial resolution of hyperspectral images. Be it through the extension of pansharpening algorithms with spatial and spectral sparsity priors, using spectral unmixing, using a low-rank assumption from complementary multisource data, or by designing an edge-preserving convex formulation, Jose Bioucas Dias set up very solid and rigourous foundations for countless subsequent works.

Owing to the powerful ability in learning low-dimensional representations and reconstruction, autoencoders (AEs) have been successfully applied in hyperspectral unmixing (HU). However, AE-based unmixing architectures, to a great extent, need to be carefully designed in a manual fashion, leading to the bulk of costs in manpower and time. To unmix hyperspectral images more intelligently, we propose an AI-powered evolvable neural architecture search method for HU, EvoNAS for short, to optimally determine the network architecture by the means of the evolutionary algorithm instead of gradient-based or reinforcement learning-based rewards. In EvoNAS, a supernet with all candidate architectures is first trained to learn the unmixing mapping in a self-supervised manner. The optimal network is then constructed by evaluating unmixing results of different architectures in the supernet. EvoNAS is capable of saving tremendous computational cost, since it inherits the weights of the pre-trained supernet and avoids training from scratch during the search phase. Experimental results conducted on two real hyperspectral datasets verify the effectiveness and superiority of the EvoNAS and show the huge potential of the NAS for HU.

With the ever-growing availability of different remote sensing (RS) products from both satellite and airborne platforms, simultaneous processing and interpretation of multimodal RS data have shown increasing significance in the RS field. Different resolutions, contexts, and sensors of multimodal RS data enable the identification and recognition of the materials lying on the earth's surface at a more accurate level by describing the same object from different points of the view. As a result, the topic on multimodal RS data fusion has gradually emerged as a hotspot research direction in recent years. This paper aims at presenting an overview of multimodal RS data fusion in several mainstream applications, which can be roughly categorized by 1) image pansharpening, 2) hyperspectral and multispectral image fusion, 3) multimodal feature learning, and (4) crossmodal feature learning. For each topic, we will briefly describe what is the to-be-addressed research problem related to multimodal RS data fusion and give the representative and state-of-the-art models from shallow to deep perspectives.

With the ever-growing availability of remote sensing (RS) data from either satellite or airborne sensors, simultaneous processing and analysis of multimodal data have been paid more and more attention by researchers in various RS-related applications. In this paper, we propose a multimodal convolutional neural network with an advanced cross-channel reconstruction module, called CCR-Net. As the name suggests, CCR-Net enables a more compact fusion of different RS data sources by the means of the reconstruction strategy across modalities that can mutually exchange information in a more effective way. Experiment are conducted on a widely-used dataset, including hyperspectral and Light Detection and Ranging (LiDAR) data, i.e., Houston2013, to verify the effectiveness and superiority of the proposed CCR - N et in comparison with several state-of-the-art baseline methods.

This paper presents a wavelet-based block low-rank representations (WBBLRR) denoising method for hyperspectral images (HSIs). WBBLRR uses 3-D wavelet transformation to decompose HSI into different blocks, where each block utilizes a low-rank representations model to obtain the denoised block, and then uses inverse 3-D wavelet transformation for all the denoised blocks to obtain the denoised HSI. The proposed method is evaluated by using both simulated and real hyperspectral datasets.

This paper presents a non-local means low-rank approximation (NLMLRA) denoising method for hyperspectral images (HSIs). NLMLRA uses a Slanted Butterworth function to construct a low-rank approximation for non-local means (NLM) operator and is efficiently implemented based on Chebyshev polynomials. The proposed method is evaluated by using both simulated and real hyperspectral datasets.

Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth "clean" hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where deep models tend to shine. In this paper, we advocate instead for a hybrid approach based on sparse coding principles that retain the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. We show on various denoising benchmarks that our method is computationally efficient and significantly outperforms the state of the art.

Over the past decades, numerous methods have been proposed to solve the linear or nonlinear mixing problems in hyperspectral unmixing (HU). The existence of spectral variabilities and nonlinearity limits, to a great extent, the unmixing ability of most traditional approaches, particularly in complex scenes. In recent years, deep learning (DL) has been garnering increasing attention in nonlinear HU owing to its powerful learning and fitting ability. However, the DL-based methods tend to generate trivial unmixing results due to the lack of considering physically meaningful endmember information. To this end, we propose a novel siamese network, called the deep half-siamese network (Deep HSNet), for HU by fully considering diverse endmember properties extracted using different endmember extraction algorithms. Moreover, the proposed Deep HSNet, beyond the previous autoencoder-like architecture, adopts another subnetwork to learn the endmember information effectively to guide the unmixing process in a reasonable and accurate way. The experimental results conducted on the synthetic and real hyperspectral data sets validate the effectiveness and superiority of the Deep HSNet over several state-of-the-art unmixing algorithms.

Light detection and ranging (LiDAR) provides high-resolution geometric information for monitoring forests at individual tree crown (ITC) level. An important task for ITC delineation is segmentation, and previous studies showed that the adaptive 3-D mean shift (AMS3D) algorithm provides effective results. AMS3D for ITC segmentation has three components for the kernel profile: shape, weight, and size. In this letter, we present an AMS3D approach based on the adaptation of the kernel profile size through an ellipsoid crown shape model. The algorithm parameters are estimated based on allometry equations derived from 22 forest plots in two study sites. After computing the mean shift (MS) vector, we initialize the parameters of the ellipsoid crown shape model to derive the kernel profile size, and further tested two crown shape models for adapting the size of the superellipsoid (SE) kernel profile. These schemes are compared with two other MS algorithms with and without kernel profile size adaptation. We select the best algorithm output per plot based on the maximum F1-score. The ellipsoid crown shape model with a SE kernel profile of$n = 1.5$presents the highest recall and the best Jaccard index, especially for conifers.

This article gives a survey of state-of-the-art methods for processing remotely sensed big data and thoroughly investigates existing parallel implementations on diverse popular high-performance computing platforms. The pros/cons of these approaches are discussed in terms of capability, scalability, reliability, and ease of use. Among existing distributed computing platforms, cloud computing is currently the most promising solution to efficient and scalable processing of remotely sensed big data due to its advanced capabilities for high-performance and service-oriented computing. We further provide an in-depth analysis of state-of-the-art cloud implementations that seek for exploiting the parallelism of distributed processing of remotely sensed big data. In particular, we study a series of scheduling algorithms (GSs) aimed at distributing the computation load across multiple cloud computing resources in an optimized manner. We conduct a thorough review of different GSs and reveal the significance of employing scheduling strategies to fully exploit parallelism during the remotely sensed big data processing flow. We present a case study on large-scale remote sensing datasets to evaluate the parallel and distributed approaches and algorithms. Evaluation results demonstrate the advanced capabilities of cloud computing in processing remotely sensed big data and the improvements in computational efficiency obtained by employing scheduling strategies.

Computational imaging for hyperspectral images (HSIs) is a hot topic in remote sensing and imaging systems. The dual-camera compressive hyperspectral imaging (DCCHI) system has been successfully designed and applied in hyperspectral imaging. However, the corresponding reconstruction algorithms are not well developed. In this paper, under the DCCHI framework, a new reconstruction algorithm is proposed based on the collaborative Tucker3 Tensor decomposition. In actual HSI, similar nonlocal patches always have similar spatial-spectral structures, and thus, these nonlocal patches can share the same spatial and spectral factors in Tucker decomposition. To characterize the similarities simultaneously, the Tucker3 decomposition is used to model the 4-order tensor formed by the similar cubic patches. To keep the spatial structures in the reconstructed HSI consistent with the panchromatic image's spatial structures, we force the spatial factor matrices and the core tensor in the Tucker3 decomposition of the HSI to be identical to the spatial factor matrices and core tensor of the panchromatic image's Tucker3 decomposition. In addition, a spectral quadratic variation constraint is introduced into the spectral factor to characterize the band smoothness. To solve the optimization problem, an alternating direction method of multipliers (ADMM)-based algorithm is designed and each variable is separately solved. Experimental results on a public data set and the remote sensing image demonstrate the advantage of the proposed method.

Conventional nonlinear subspace learning techniques (e.g., manifold learning) usually introduce some drawbacks in explainability (explicit mapping) and cost effectiveness (linearization), generalization capability (out-of-sample), and representability (spatial-spectral discrimination). To overcome these shortcomings, a novel linearized subspace analysis technique with spatial-spectral manifold alignment is developed for a semisupervised hyperspectral dimensionality reduction (HDR), called joint and progressive subspace analysis (JPSA). The JPSA learns a high-level, semantically meaningful, joint spatial-spectral feature representation from hyperspectral (HS) data by: 1) jointly learning latent subspaces and a linear classifier to find an effective projection direction favorable for classification; 2) progressively searching several intermediate states of subspaces to approach an optimal mapping from the original space to a potential more discriminative subspace; and 3) spatially and spectrally aligning a manifold structure in each learned latent subspace in order to preserve the same or similar topological property between the compressed data and the original data. A simple but effective classifier, that is, nearest neighbor (NN), is explored as a potential application for validating the algorithm performance of different HDR approaches. Extensive experiments are conducted to demonstrate the superiority and effectiveness of the proposed JPSA on two widely used HS datasets: 1) Indian Pines (92.98%) and 2) the University of Houston (86.09%) in comparison with previous state-of-the-art HDR methods. The demo of this basic work (i.e., ECCV2018) is openly available at https://github.com/danfenghong/ECCV2018_J-Play.

Remote sensing data from hyperspectral cameras suffer from limited spatial resolution, in which a single pixel of a hyperspectral image may contain information from several materials in the field of view. Blind hyperspectral image unmixing is the process of identifying the pure spectra of individual materials (i.e., endmembers) and their proportions (i.e., abundances) at each pixel. In this article, we propose a novel blind hyperspectral unmixing model based on the graph total variation (gTV) regularization, which can be solved efficiently by the alternating direction method of multipliers (ADMM). To further alleviate the computational cost, we apply the Nyström method to approximate a fully connected graph by a small subset of sampled points. Furthermore, we adopt the Merriman-Bence-Osher (MBO) scheme to solve the gTV-involved subproblem in ADMM by decomposing a gray-scale image into a bitwise form. A variety of numerical experiments on synthetic and real hyperspectral images are conducted, showcasing the potential of the proposed method in terms of identification accuracy and computational efficiency.

Although multimodal remote sensing data analysis can strongly improve the characterization of physical phenomena on Earth's surface, nonidealities and estimation imperfections between records and investigation models can limit its actual information extraction ability. In this article, we aim at predicting the maximum information extraction that can be reached when analyzing a given data set. By means of an asymptotic information theory-based approach, we investigate the reliability and accuracy that can be achieved under optimal conditions for multimodal analysis as a function of data statistics and parameters that characterize the multimodal scenario to be addressed. Our approach leads to the definition of two indices that can be easily computed before the actual processing takes place. Moreover, we report in this article how they can be used for operational use in terms of image selection in order to maximize the robustness of the multimodal analysis, as well as to properly design data collection campaigns for understanding and quantifying physical phenomena. Experimental results show the consistency of our approach.

The fusion of high spatial resolution panchromatic (PAN) data with simultaneously acquired multispectral (MS) data with the lower spatial resolution is a hot topic, which is often called pansharpening. In this article, we exploit the combination of machine learning techniques and fusion schemes introduced to address the pansharpening problem. In particular, deep convolutional neural networks (DCNNs) are proposed to solve this issue. The latter is combined first with the traditional component substitution and multiresolution analysis fusion schemes in order to estimate the nonlinear injection models that rule the combination of the upsampled low-resolution MS image with the extracted details exploiting the two philosophies. Furthermore, inspired by these two approaches, we also developed another DCNN for pansharpening. This is fed by the direct difference between the PAN image and the upsampled low-resolution MS image. Extensive experiments conducted both at reduced and full resolutions demonstrate that this latter convolutional neural network outperforms both the other detail injection-based proposals and several state-of-the-art pansharpening methods.

Extensive attention has been widely paid to enhance the spatial resolution of hyperspectral (HS) images with the aid of multispectral (MS) images in remote sensing. However, the ability in the fusion of HS and MS images remains to be improved, particularly in large-scale scenes, due to the limited acquisition of HS images. Alternatively, we super-resolve MS images in the spectral domain by the means of partially overlapped HS images, yielding a novel and promising topic: spectral superresolution (SSR) of MS imagery. This is challenging and less investigated task due to its high ill-posedness in inverse imaging. To this end, we develop a simple but effective method, called joint sparse and low-rank learning (J-SLoL), to spectrally enhance MS images by jointly learning low-rank HS-MS dictionary pairs from overlapped regions. J-SLoL infers and recovers the unknown HS signals over a larger coverage by sparse coding on the learned dictionary pair. Furthermore, we validate the SSR performance on three HS-MS data sets (two for classification and one for unmixing) in terms of reconstruction, classification, and unmixing by comparing with several existing state-of-the-art baselines, showing the effectiveness and superiority of the proposed J-SLoL algorithm. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_J-SLoL, contributing to the remote sensing (RS) community.

Classification and identification of the materials lying over or beneath the earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS), and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML)-cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on “what,” “where,” and “how” to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS data sets. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

Convolutional neural networks (CNNs) have been attracting increasing attention in hyperspectral (HS) image classification due to their ability to capture spatial-spectral feature representations. Nevertheless, their ability in modeling relations between the samples remains limited. Beyond the limitations of grid sampling, graph convolutional networks (GCNs) have been recently proposed and successfully applied in irregular (or nongrid) data representation and analysis. In this article, we thoroughly investigate CNNs and GCNs (qualitatively and quantitatively) in terms of HS image classification. Due to the construction of the adjacency matrix on all the data, traditional GCNs usually suffer from a huge computational cost, particularly in large-scale remote sensing (RS) problems. To this end, we develop a new minibatch GCN (called miniGCN hereinafter), which allows to train large-scale GCNs in a minibatch fashion. More significantly, our miniGCN is capable of inferring out-of-sample data without retraining networks and improving classification performance. Furthermore, as CNNs and GCNs can extract different types of HS features, an intuitive solution to break the performance bottleneck of a single model is to fuse them. Since miniGCNs can perform batchwise network training (enabling the combination of CNNs and GCNs), we explore three fusion strategies: additive fusion, elementwise multiplicative fusion, and concatenation fusion to measure the obtained performance gain. Extensive experiments, conducted on three HS data sets, demonstrate the advantages of miniGCNs over GCNs and the superiority of the tested fusion strategies with regard to the single CNN or GCN models. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_GCN for the sake of reproducibility.

In this article, we have great interest in investigating a common but practical issue in remote sensing (RS)-can a limited amount of one information-rich (or high-quality) data, e.g., hyperspectral (HS) image, improve the performance of a classification task using a large amount of another information-poor (low-quality) data, e.g., multispectral (MS) image? This question leads to a typical cross-modality feature learning. However, classic cross-modality representation learning approaches, e.g., manifold alignment, remain limited in effectively and efficiently handling such problems that the data from high-quality modality are largely absent. For this reason, we propose a novel graph-induced aligned learning (GiAL) framework by 1) adaptively learning a unified graph (further yielding a Laplacian matrix) from the data in order to align multimodality data (MS-HS data) into a latent shared subspace; 2) simultaneously modeling two regression behaviors with respect to labels and pseudo-labels under a multitask learning paradigm; and 3) dramatically updating the pseudo-labels according to the learned graph and refeeding the latest pseudo-labels into model learning of the next round. In addition, an optimization framework based on the alternating direction method of multipliers (ADMMs) is devised to solve the proposed GiAL model. Extensive experiments are conducted on two MS-HS RS data sets, demonstrating the superiority of the proposed GiAL compared with several state-of-the-art methods.

This article addresses the problem of semantic segmentation with limited cross-modality data in large-scale urban scenes. Most prior works have attempted to address this issue by using multimodal deep neural networks (DNNs). However, their ability to effectively blending different properties across multimodalities and robustly learning representations from complex scenes remains limited, particularly in the absence of sufficient and well-annotated training images. This leads to a challenge related to cross-modality learning with multimodal DNNs. To this end, we introduce two novel plug-and-play units in the network: self-generative adversarial networks (GANs) module and mutual-GANs module, to learn perturbation-insensitive feature representations and to eliminate the gap between multimodalities, respectively, yielding more effective and robust information transfer. Furthermore, a patchwise progressive training strategy is devised to enable effective network learning with limited samples. We evaluate the proposed network on two multimodal (hyperspectral and multispectral) overhead image data sets and achieve a significant improvement in comparison with several state-of-the-art methods.

In the past few years, graph-based methods have proven to be a useful tool in a wide variety of energy minimization problems. In this article, we propose a graph-based algorithm for feature extraction and segmentation of multimodal images. By defining a notion of similarity that integrates information from each modality, we create a fused graph that merges the different data sources. The graph Laplacian then allows us to perform feature extraction and segmentation on the fused data set. We apply this method in a practical example, namely, the segmentation of optical and LiDAR images. The results obtained confirm the potential of the proposed method.

Due to the proliferation of large-scale remote-sensing (RS) archives with multiple annotations, multilabel RS scene classification and retrieval are becoming increasingly popular. Although some recent deep learning-based methods are able to achieve promising results in this context, the lack of research on how to learn embedding spaces under the multilabel assumption often makes these models unable to preserve complex semantic relations pervading aerial scenes, which is an important limitation in RS applications. To fill this gap, we propose a new graph relation network (GRN) for multilabel RS scene categorization. Our GRN is able to model the relations between samples (or scenes) by making use of a graph structure which is fed into network learning. For this purpose, we define a new loss function called scalable neighbor discriminative loss with binary cross entropy (SNDL-BCE) that is able to embed the graph structures through the networks more effectively. The proposed approach can guide deep learning techniques (such as convolutional neural networks) to a more discriminative metric space, where semantically similar RS scenes are closely embedded and dissimilar images are separated from a novel multilabel viewpoint. To achieve this goal, our GRN jointly maximizes a weighted leave-one-out K-nearest neighbors ( KNN) score in the training set, where the weight matrix describes the contributions of the nearest neighbors associated with each RS image on its class decision, and the likelihood of the class discrimination in the multilabel scenario. An extensive experimental comparison, conducted on three multilabel RS scene data archives, validates the effectiveness of the proposed GRN in terms of KNN classification and image retrieval. The codes of this article will be made publicly available for reproducible research in the community.

Tensor-based fusion that couples the high spatial resolution of a multispectral image (MSI) to the high spectral resolution of a hyperspectral image (HSI) is considered. The fusion problem is first formulated mathematically as a convex optimization of a tensor trace norm imposing low-rank spatially as well as spectrally, with an alternating-directions optimization featuring linearization providing the solution. Although prior tensor-based fusion approaches typically resort to tensor decomposition, the proposed algorithm exploits ideas from the field of tensor completion to directly impose a low-rank property spatially and spectrally while avoiding the computationally complex patch clustering and dictionary learning common to competing fusion techniques. Additionally, small modifications to the basic optimization permit a fusion process robust to missing hyperspectral values such as those that can result from dead stripes in real hyperspectral sensors. The experimental evaluations on both synthetic imagery as well as real imagery demonstrate that the resulting low-rank tensor-approximation (LRTA) fusion algorithm preserves both spatial details and texture, yielding significantly improved image quality when compared to other state-of-the-art fusion methods as well as effective restoration under conditions of missing stripes within the HSI.

Recently, deep learning methods have attracted much attention in the field of polarimetric synthetic aperture radar (PolSAR) data interpretation and understanding. However, for supervised methods, it requires large-scale labeled data to achieve better performance, and getting enough labeled data is a time-consuming and laborious task. Aiming to obtain a good classification result with limited labeled data, we focus on learning discriminative high-level features between multiple representations, which we call mutual information. As PolSAR data have multi-modal representations, there should have strong similarity between multi-modal features of the same pixel. In addition, each pixel has its own unique geocoding and scattering information. Hence, every pixel has great difference from other pixels in a specific representation space. Based on the above observations, this article proposes a mutual information-based self-supervised learning (MI-SSL) model to learn an implicit representation from unlabeled data. In this article, the self-supervised learning idea is first applied to PolSAR data processing. Furthermore, a reasonable pretext task, which is suitable for PolSAR data, is designed to extract mutual information for classification tasks. Compared with the state-of-the-art classification methods, experimental results on four PolSAR data sets demonstrate that our MI-SSL model produces impressive overall accuracy with fewer labeled data.

The use of hyperspectral (HS) data is growing over the years, thanks to the very high spectral resolution. However, HS data are still characterized by a spatial resolution that is too low for several applications, thus motivating the design of fusion techniques aimed to sharpen HS images with high spatial resolution data. To reach a significant resolution enhancement, high-resolution images should be acquired by different satellite platforms. In this article, we highlight the pros and cons of employing real multiplatform data, using the EO-1 satellite as an exemplary case. The spatial resolution of the HS data collected by the Hyperion sensor is improved by exploiting both the ALI panchromatic image collected from the same platform and acquisitions from the WorldView-3 and the QuickBird satellites. Furthermore, we tackle the problem of assessing the final quality of the fused product at the nominal resolution, which presents further difficulties in this general environment. Useful indications for the design of an effective sharpening method in this case are finally outlined.

The total variation (TV) regularization has been widely used in various applications related to hyperspectral (HS) signal and image processing due to its potential in modeling the underlying smoothness of HS data. However, most existing TV norms usually tend to generate spatial oversmoothing or artifacts. To this end, we propose a novel l0- l1hybrid TV ( l0- l1HTV) regularization with the applications to HS mixed noise removal and compressed sensing (CS). More specifically, l0- l1HTV can be regarded as a globally and locally integrated TV regularizer, where the l0gradient constraint is incorporate into the l1spatial-spectral TV ( l1-SSTV). l1-SSTV is capable of exploiting the local structure information across both spatial and spectral domains, while the l0gradient can promote a globally spectral-spatial smoothness by directly controlling the number of nonzero gradients of HS images. This efficient combination considers more comprehensive prior knowledge of HS images, yielding sharper edge preservation and resolving the above drawbacks of existing pure TV norms. More significantly, l0- l1HTV can be easily injected into HS-related processing models, and an effective algorithm based on the alternating direction method of multipliers (ADMM) is developed to solve the optimization problems. Extensive experiments conducted on several HS data sets substantiate the superiority and effectiveness of the proposed method in comparison with many state-of-the-art methods.

Conventional low-rank (LR)-based hyperspectral image (HSI) denoising models generally convert high-dimensional data into 2-D matrices or just treat this type of data as 3-D tensors. However, these pure LR or tensor low-rank (TLR)-based methods lack flexibility for considering different correlation information from different HSI directions, which leads to the loss of comprehensive structure information and inherent spatial-spectral relationship. To overcome these shortcomings, we propose a novel multidirectional LR modeling and spatial-spectral total variation (MLR-SSTV) model for removing HSI mixed noise. By incorporating the weighted nuclear norm, we obtain the weighted sum of weighted nuclear norm minimization (WSWNNM) and the weighted sum of weighted tensor nuclear norm minimization (WSWTNNM) to estimate the more accurate LR tensor, especially, to remove the dead-line noise better. Gaussian noise is further denoised and the local spatial-spectral smoothness is preserved effectively by SSTV regularization. We develop an efficient algorithm for solving the derived optimization based on the alternating direction method of multipliers (ADMM). Extensive experiments on both synthetic data and real data demonstrate the superior performance of the proposed MLR-SSTV model for HSI mixed noise removal.

Due to the limitations of hyperspectral imaging systems, hyperspectral imagery (HSI) often suffers from poor spatial resolution, thus hampering many applications of the imagery. Hyperspectral super resolution refers to fusing HSI and MSI to generate an image with both high spatial and high spectral resolutions. Recently, several new methods have been proposed to solve this fusion problem, and most of these methods assume that the prior information of the point spread function (PSF) and spectral response function (SRF) are known. However, in practice, this information is often limited or unavailable. In this work, an unsupervised deep learning-based fusion method-HyCoNet-that can solve the problems in HSI-MSI fusion without the prior PSF and SRF information is proposed. HyCoNet consists of three coupled autoencoder nets in which the HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model. Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process. Furthermore, driven by the joint loss function, the proposed method is straightforward and easily implemented in an end-to-end training manner. The experiments performed in the study demonstrate that the proposed method performs well and produces robust results for different data sets and arbitrary PSFs and SRFs.

Deep convolutional neural networks attract increasing attention in image patch matching. However, most of them rely on a single similarity learning model, such as feature distance and the correlation of concatenated features. Their performances will degenerate due to the complex relation between matching patches caused by various imagery changes. To tackle this challenge, we propose a multi-relation attention learning network (MRAN) for image patch matching. Specifically, we propose to fuse multiple feature relations (MR) for matching, which can benefit from the complementary advantages between different feature relations and achieve significant improvements on matching tasks. Furthermore, we propose a relation attention learning module to learn the fused relation adaptively. With this module, meaningful feature relations are emphasized and the others are suppressed. Extensive experiments show that our MRAN achieves best matching performances, and has good generalization on multi-modal image patch matching, multi-modal remote sensing image patch matching and image retrieval tasks.

In this paper, the polarimetric synthetic aperture radar (PolSAR) scene classification is solved by using a novel low -rank tensor-based multi-view subspace representation (LRT-MSR) method. PolSAR data can be described in multimodal feature spaces, such as PolSAR coherent/covariance/scattering matrices, or the various polarimetric decompositions. Different pseudo-color images from multiple spaces provide enough visual information for making a comprehensive representation. Our method applies a low-rank tensor-based subspace clustering way to explore the information from multi-view pseudo-color images. Tensor, as the high order matrix, is used to capture the correlations of underlying multi-view data. Furthermore, the method is constrained by a low-rank term that elegantly models the cross information from different views, and achieves a series of representation matrices from the redundant information. Finally, a spectral cluster method is used to make the final classification. The experimental results on PolSAR image dataset show the effectiveness of the applied method.

This work presents a novel hyperspectral embedding technique by learning a deep regression network in an unsupervised fashion, which aims at reducing the computational complexity and storage-costing of traditional manifold embedding methods as well as improving the representation ability of spectral signatures effectively. The proposed method attempts to learn an explicit and unified nonlinear mapping from all patch-wise correspondences of original hyperspectral data and dimension-reduced products generated by some existing manifold learning approaches. This process can be well performed by means of a deep regression model. The learned model is not only capable of locally capturing the manifold structure of the whole hyperspectral image from densely patch-based random sampling but also better applicable to high-efficient out-of-sample inference. Experimental results conducted on the real hyperspectral data demonstrate the effectiveness and superiority of the proposed hyperspectral embedding technique.

Recently, spectral enhancement of multispectral imagery has attracted a growing interest in the remote sensing (RS) community. Without any prior knowledge, this task is highly ill-conditioned in inverse problems. To this end, we develop a simple but effective method, called locally linear reconstruction (LLR), to spectrally enhance the multispectral imagery (MSI) using partially overlapped hyperspectral data. LLR learns reconstruction coefficients of each pixel from the MSI and shares the same weights to recover the unknown hyperspectral signals over a larger coverage. We validate the performance of the proposed LLR on the real hyperspectral data in comparison with several state-of-the-art baselines, demonstrating its effectiveness and superiority.

This paper presents a local spatial-spectral correlation based mixtures of factor analyzers (LSSC-MFA) denoising method for hyperspectral image (HSI). HSIs are usually degraded by different noise types such as missing lines (ML), missing pixels (MP), salt and pepper noise (SP), and Gaussian noise. The proposed method, hierarchically, removes the mixed noise. Firstly, we develop a novel local spatial-spectral correlation (LSSC) method to remove the ML noise. Then LSSC-MFA uses the mixtures of factor analyzers (MFA) method to remove the MP, SP, and Gaussian noises. The performance of the proposed method has been validated using both real and simulated HSI datasets. Results on the simulated datasets confirm considerable improvements in terms of peak signal-to-noise ratio (PSNR) compared to the state-of-the-art denoising methods used in experiments. In addition, visual improvements can be observed in the case of real dataset experiments.

This paper presents two hyperspectral image (HSI) denoising methods, mixtures of factor analyzers (MFA) and wavelet-based MFA (WMFA). MFA uses a Gaussian mixture model to segment the original HSI into different parts, where each part follows Gaussian distribution and then utilizes a factor analyzer to get a low-rank factor loading matrix, and finally uses the inverse transformation of the matrix to get the denoised hyperspectral dataset. WMFA uses the MFA in the wavelet domain to remove the noise in HSI. The proposed methods are evaluated by using both simulated and real hyperspectral datasets.

In hyperspectral imaging, spectral unmixing aims at decomposing the image into a set of reference spectral signatures corresponding to the materials present in the observed scene and their relative proportions in every pixel. While a linear mixing model was used for a long time, the complex nature of the physicochemical phenomena that affect the spectra of the materials led to shift the community's attention toward algorithms accounting for the variability of the endmembers. Such intraclass variations are mainly due to local changes in the composition of the materials and to illumination changes. In the physical remote sensing community, a popular model accounting for illumination variability is the radiative transfer model proposed by Hapke. It is, however, too complex to be directly used in hyperspectral unmixing in a tractable way. Instead, the extended linear mixing model (ELMM) allows to easily unmix the hyperspectral data accounting for changing illumination conditions and to address nonlinear effects to some extent. In this letter, we show that the ELMM can be obtained from the Hapke model by successively simplifying physical assumptions, whose validity we experimentally examine, thus demonstrating its relevance to handle illumination-induced variability in the unmixing problem.

Due to the ever-growing diversity of the data source, multimodality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multimodalities that exist in both training and test sets, yet they are less investigated in the absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multimodalities in the training process. By this way, the out-of-sample from any of multimodalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multimodal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Light Detection and Ranging (LIDAR) and hyperspectral data set provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.

This letter proposes a novel method for combining multiview features in polarimetric synthetic aperture radar (PolSAR) for land cover classification. It is well-known that feature extraction and classifier design are two significant steps in machine learning methods for PolSAR data interpretation. Each PolSAR pixel can be represented in different feature spaces, such as polarimetric data scattering, or the polarimetric target decomposition spaces. In this letter, a tensor-based multiview embedding algorithm is proposed to fuse those features from different spaces in order to obtain a distinctive set of features for the subsequent classification. Based on the pixel-based classification tasks, a modified tensor distance (MTD) is designed to accurately calculate the distance between tensors. It emphasizes the importance of the central pixel, and decreases the influence of the neighbors in the feature patch when calculating tensor distance. Furthermore, the complementary properties of different views are exploited by an MTD measured tensor multiview spectral embedding method, so as to obtain relevant low-dimensional features. Compared with state-of-the-art methods, the validation and effectiveness of the proposed method is demonstrated on two real PolSAR data sets.

Pansharpening techniques allow a detailed reproduction of the Earth surface by fusing a multispectral (MS) and a panchromatic (PAN) image acquired over the same area. Classical pansharpening methods consist in the extraction of the details from the PAN image and their subsequent injection into the MS image through a linear function. In this letter, we propose to apply a nonlinear injection procedure that implements the detail injection through a polynomial function. Optimal polynomial coefficients in the least squares sense can be easily obtained in the closed form, and the consequent pansharpening algorithm is shown to obtain superior performance with respect to the existing linear approaches, especially for MS bands with a reduced wavelength overlap with the PAN channel.

Geospatial object detection (GOD) of remote sensing imagery has been attracting increasing interest in recent years, due to the rapid development in spaceborne imaging. Most of the previously proposed object detectors are very sensitive to object deformations, such as scaling and rotation. To this end, we propose a novel and efficient framework for GOD in this letter, called Fourier-based rotation-invariant feature boosting (FRIFB). A Fourier-based rotation-invariant feature is first generated in polar coordinate. Then, the extracted features can be further structurally refined using aggregate channel features. This leads to a faster feature computation and more robust feature representation, which is good fitting for the coming boosting learning. Finally, in the test phase, we achieve a fast pyramid feature extraction by estimating a scale factor instead of directly collecting all features from the image pyramid. Extensive experiments are conducted on two subsets of NWPU VHR-10 data set, demonstrating the superiority and effectiveness of the FRIFB compared to the previous state-of-the-art methods.

Single-image super-resolution (SISR) techniques attempt to reconstruct the finer resolution version of a given image from its coarser version. In the SISR of hyperspectral data sets, the simultaneous consideration of spectral bands is crucial for ensuring the spectral fidelity. However, the high spectral resolution of these data sets affects the performance of conventional approaches. This research proposes the design of 3-D convolutional neural network (CNN)-based SISR architectures that can map the spatial-spectral characteristics of hypercubes to a finer spatial resolution. The proposed approaches facilitate the simultaneous optimization of sparse codes and dictionaries with regard to the super-resolution objective. Our main hypothesis is that the consideration of spectral aspects is essential for the spatial enhancement of hyperspectral images. Also, we propose that the regularized deconvolution of a coarser-scale hypercube, using learned 3-D filters, yields the required high-resolution version. Based on these hypotheses, a convolution-deconvolution framework is proposed to super-resolve the hypercubes in parallel with the reconstruction of a set of regularizing features. Novel sparse code optimization sub-networks proposed in this article give better performance than the existing strategies. The endmember similarities and hyperspectral image prior are considered while designing the proposed loss functions. In order to improve the generalizability, a collaborative spectral unmixing strategy is employed to refine the spectral base of the super-resolved result. The spatial-spectral accuracy of the super-resolved hypercubes, in terms of the validity of regularizing features and endmembers, is explored to devise an optimal ensemble strategy. The experiments, over different data sets, confirm better accuracy of the proposed frameworks compared to the prominent approaches.

So far, a large number of advanced techniques have been developed to enhance and extract the spatially semantic information in hyperspectral image processing and analysis. However, locally semantic change, such as scene composition, relative position between objects, spectral variability caused by illumination, atmospheric effects, and material mixture, has been less frequently investigated in modeling spatial information. Consequently, identifying the same materials from spatially different scenes or positions can be difficult. In this article, we propose a solution to address this issue by locally extracting invariant features from hyperspectral imagery (HSI) in both spatial and frequency domains, using a method called invariant attribute profiles (IAPs). IAPs extract the spatial invariant features by exploiting isotropic filter banks or convolutional kernels on HSI and spatial aggregation techniques (e.g., superpixel segmentation) in the Cartesian coordinate system. Furthermore, they model invariant behaviors (e.g., shift, rotation) by the means of a continuous histogram of oriented gradients constructed in a Fourier polar coordinate. This yields a combinatorial representation of spatial-frequency invariant features with application to HSI classification. Extensive experiments conducted on three promising hyperspectral data sets (Houston2013 and Houston2018) to demonstrate the superiority and effectiveness of the proposed IAP method in comparison with several state-of-the-art profile-related techniques. The codes will be available from the website: https://sites.google.com/view/danfeng-hong/data-code.

Polarimetric synthetic aperture radar (PolSAR) as a typical multi-channel sensor can obtain refined geometrical and geophysical information. In the PolSAR land cover classification task, feature extraction is regarded as a critical step for the final classification. It can employ multi-modal features from the original scattering data, polarimetric target decomposition, and other transformation space. Then, how to efficiently combine multi-modal polarimetric information and extract discriminant features is an important challenge for PolSAR image processing. Graph embedding methods have become a significant technique to deal with feature extraction and dimensionality reduction (DR) problems in recent years. It provides a unified linearization framework in machine learning and other pattern recognition tasks. In this article, an extended tensor embedding framework is introduced to extract the intrinsic features for PolSAR land cover classification. First, each pixel is represented by a feature cube that is constructed by groups of polarimetric scattering signals and target decomposition features in a fixed size patch. Second, an intrinsic matrix is constructed to describe the original geometrical and statistical properties of the samples, and a penalty matrix is designed to represent some constraints. Third, the vector-based algorithms are transformed into tensor space in an unified framework and based on the pair of matrices to obtain the projection matrices in each mode by an iterative optimization process. The effectiveness of the proposed methods is demonstrated on three RADARSAT2 data sets covering the regions of Xi'an, San Francisco, and Flevoland, respectively. The visualization and quantification results show that the proposed method has superiority in land cover classification.

Pansharpening refers to the combination of panchromatic (PAN) and multispectral (MS) images, designed to obtain a fused product retaining the fine spatial resolution of the former and the high spectral content of the latter. One of the most popular and successful approaches to pansharpening is the method known as context-based generalized Laplacian pyramid, which requires as a key ingredient for the estimation of the so-called injection coefficients. In this article, we propose the adoption of robust techniques for the estimation of the injection coefficients and detection strategies to select the clusters for which robust regression is needed, providing a suitable balancing between fusion performance and computational burden. Experimental results conducted on five real data sets acquired by the sensors QuickBird, WorldView-3, and WorldView-4, show the superiority of the proposed method with respect to current state-of-the-art pansharpening techniques.

Hyperspectral (HS) super-resolution, which aims at enhancing the spatial resolution of hyperspectral images (HSIs), has recently attracted considerable attention. A common way of HS super-resolution is to fuse the HSI with a higher spatial-resolution multispectral image (MSI). Various approaches have been proposed to solve this problem by establishing the degradation model of low spatial-resolution HSIs and MSIs based on matrix factorization methods, e.g., unmixing and sparse representation. However, this category of approaches cannot well construct the relationship between the high-spatial-resolution (HR) HSI and MSI. In fact, since the HSI and the MSI capture the same scene, these two image sources must have common factors. In this paper, a nonlocal tensor decomposition model for hyperspectral and multispectral image fusion (HSI-MSI fusion) is proposed. First, the nonlocal similar patch tensors of the HSI are constructed according to the MSI for the purpose of calculating the smooth order of all the patches for clustering. Then, the relationship between the HR HSI and the MSI is explored through coupled tensor canonical polyadic (CP) decomposition. The fundamental idea of the proposed model is that the factor matrices in the CP decomposition of the HR HSI's nonlocal tensor can be shared with the matrices factorized by the MSI's nonlocal tensor. Alternating direction method of multipliers is used to solve the proposed model. Through this method, the spatial structure of the MSI can be successfully transferred to the HSI. Experimental results on three synthetic data sets and one real data set suggest that the proposed method substantially outperforms the existing state-of-the-art HSI-MSI fusion methods.

Spectral-spatial framework has been widely applied for hyperspectral image classification task. Some well-established models, such as group sparse representation (GSR), have gained a certain advance but still mainly focus on the usage of local spatial similarity and neglect the nonlocal spatial information. Recently, nonlocal self-similarity (NLSS) has been exploited to support the spatial coherence tasks. However, current NLSS-based methods are biased toward the direct use of nonlocal spatial information as a whole, while the underlying spectral information is not well exploited. In this article, we proposed a novel method to exploit local spectral similarity through nonlocal spatial similarity, with the integration of local spatial consistency in a single framework. Specifically, the proposed approach first exploits the NLSS by searching the nonoverlapped similar patches in defined scopes. Then, spectral similarity is determined locally within the found patches. After that, the found similar data and the original data are fused in a designed pattern. Finally, the GSR-based classifier (GSRC) is applied to process the fused data characterized by the manifold learning algorithm. The experimental results based on three real hyperspectral data sets demonstrate the efficiency of the proposed method, with improvements over the other related nonlocal or local similarity-based methods.

Inspired by the recent developments in computer vision, low-rank and structured sparse matrix decomposition can be potentially be used for extract moving objects in satellite videos. This set of approaches seeks for rank minimization on the background that typically requires batch-based optimization over a sequence of frames, which causes delays in processing and limits their applications. To remedy this delay, we propose an online low-rank and structured sparse decomposition (O-LSD). O-LSD reformulates the batch-based low-rank matrix decomposition with the structured sparse penalty to its equivalent framewise separable counterpart, which then defines a stochastic optimization problem for online subspace basis estimation. In order to promote online processing, O-LSD conducts the foreground and background separations and the subspace basis update alternatingly for every frame in a video. We also show the convergence of O-LSD theoretically. Experimental results on two satellite videos demonstrate the performance of O-LSD in terms of accuracy, and the time consumption is comparable with the batch-based approaches with significantly reduced delay in processing.

Convolutional neural networks (CNNs) have recently achieved impressive improvements on hyperspectral (HS) pansharpening. However, most of the CNN-based HS pansharpening approaches would have to first upsample the low-resolution hyperspectral image (LR-HSI) using bicubic interpolation or data-driven training strategy, which inevitably lose some details or greatly rely on the learning process. In addition, most previous methods regard the pansharpening as a black-box problem and treat diverse features equally, thus hindering the discriminative ability of CNNs. To conquer these issues, a novel HS pansharpening method using deep hyperspectral prior (DHP) and dual-attention residual network (DARN) is proposed in this article. Specifically, we first upsample the LR-HSI to the scale of the panchromatic (PAN) image through the DHP algorithm, which can better preserve spatial and spectral information without learning from large data sets. The upsampled result is then concatenated with the PAN image to form the input of the DARN, where several channel-spatial attention residual blocks (CSA ResBlocks) are stacked to map the residual HSI between the reference HSI and the upsampled HSI. In each CSA ResBlock, two complementary attention modules, i.e., channel attention and spatial attention modules, are designed to adaptively learn more informative features of spectral channels and spatial locations simultaneously, which can effectively boost the fusion accuracy. Finally, the fused HSI is obtained by the summation of the upsampled HSI and the reconstructed residual HSI. The experimental results of both simulated and real HS data sets demonstrate that the performance of our DHP-DARN method is superior over the state-of-the-art HS pansharpening approaches.

Image fusion is growing interest in recent years, thanks to the huge amount of data acquired everyday by sensors on board of satellite platforms. The enhancement of the spatial resolution of a multispectral (MS) image through the use of a panchromatic (PAN) image, usually called pansharpening, is getting more and more relevant. In this work, we focus on the problem of the estimation of the injection coefficients that rule the enhancement of the spatial resolution of the MS image by properly adding the PAN details. In particular, a statistical analysis of the residuals coming from the linear multivariate regression between details extracted from the PAN image and the MS image is performed. A novel hybrid model is introduced for accurately describing the statistical distribution of these residuals, together with a procedure for efficiently estimating both the parameters of the residual distribution and the injection coefficients. The improvements achieved by the proposed approach are assessed using two very high resolution datasets acquired by the WorldView-3 and Worldview-4 satellites. The benefits of the proposed approach are particularly clear when vegetated areas are involved in the fusion process.

Hyperspectral image unmixing has proven to be a useful technique to interpret hyperspectral data, and is a prolific research topic in the community. Most of the approaches used to perform linear unmixing are based on convex geometry concepts, because of the strong geometrical structure of the linear mixing model. However, many algorithms based on convex geometry are still used in spite of the underlying model not considering the intra-class variability of the materials. A natural question is to wonder to what extent these concepts and tools (Intrinsic Dimensionality estimation, endmember extraction algorithms, pixel purity) are still relevant when spectral variability comes into play. In this paper, we first analyze their robustness in a case where the linear mixing model holds in each pixel, but the endmembers vary in each pixel according to a prescribed variability model. In the light of this analysis, we propose an integrated unmixing chain which tries to adress the shortcomings of the classical tools used in the linear case, based on our previously proposed extended linear mixing model. We show the interest of the proposed approach on simulated and real datasets.

Hyperspectral image (HSI) super-resolution is a hot topic in remote sensing and computer vision. Recently, tensor analysis has been proven to be an efficient technology for HSI image processing. However, the existing tensor-based methods of HSI super-resolution are not able to capture the high-order correlations in HSI. In this article, we propose to learn a high-order coupled tensor ring (TR) representation for HSI super-resolution. The proposed method first tensorizes the HSI to be estimated into a high-order tensor in which multiscale spatial structures and the original spectral structure are represented. Then, a coupled TR representation model is proposed to fuse the low-resolution HSI (LR-HSI) and high-resolution multispectral image (HR-MSI). In the proposed model, some latent core tensors in TR of the LR-HSI and the HR-MSI are shared, and we use the relationship between the spectral core tensors to reconstruct the HSI. In addition, the graph-Laplacian regularization is introduced to the spectral core tensors to preserve the spectral information. To enhance the robustness of the proposed model, Frobenius norm regularizations are introduced to the other core tensors. Experimental results on both synthetic and real data sets show that the proposed method achieves the state-of-the-art super-resolution performance.

Recently, enormous efforts have been made to improve the performance of the linear or nonlinear mixing model for hyperspectral unmixing, yet their ability to handle spectral variability and extract physically meaningful endmembers remains limited. Based on the powerful learning ability of deep learning, we propose a weakly-supervised unmixing network, called WU-Net, to break the bottleneck. Beyond the autoencoder-like architecture, WU-Net learns an additional network from the pure or nearly-pure endmembers to correct the weights of another unmixing network towards a more accurate and interpretable unmixing solution, thus yielding a two-stream deep network. Experimental results conducted on two different datasets, one fully artificial simulation dataset and one simulated EnMap dataset generated from a real HyMap dataset, demonstrate the effectiveness and superiority of WU-Net over several state-of-the-art algorithms.

In recent years, there have been many advances in sensor technology which provides more useful information about the observed scene. Two of the newest remote sensing technologies are hyperspectral (HS) and Light Detection And Ranging (LiDAR) sensors. Since pixels in a small spatial neighborhood are more likely to share similar abundances, hypergraph regularization (HG-NMF) can be employed to handle the similarity relevance among the spatial neighborhood pixels. In this paper, we provide a LiDAR data-aided HS unmixing using HG-NMF. The composite usage of all these valuable information can lead to higher accuracy unmixing results. The obtained convex optimization problem is solved by Spectral Unmixing by Split Augmented Lagrangian (SUnSAL-TV) Algorithm. Experiments on synthetic data are conducted. The advantage of HG-NMF regularization is also demonstrated.

This paper aims to tackle a general but interesting cross-modality problem in remote sensing community: can multimodal images help to generate synthetic images in time series and improve temporal resolution? To this end, we explore multimodal-temporal fusion, in which we attempt to leverage the availability of additional cross-modality images to simulate the missing images in time series. We propose a multimodal-temporal fusion framework, and mainly focus on two kinds of information for the simulation: intra-modal cross-modality information and inter-modal temporal information. To exploit the cross-modality information, we adopt available paired images and learn a mapping between different modality images using a deep neural network. Considering temporal dependency among time-series images, we formulate a temporal constraint in the learning to encourage temporal consistent results. Experiments are conducted on two cross-modality image simulation applications (SAR to visible and visible to SWIR), and both visual and quantitative results demonstrate that the proposed model can successfully simulate missing images with cross-modality data.

Flash LiDARs have been described as key components for future space missions including rover operations and the descent and landing of spacecraft. The system records full 3D-images along with 2D intensity images with a single laser pulse, thus eliminating the need for a scanning device. There is a vast literature and well-established techniques concerning radiometric calibration (2D information). However, the illumination of the target is usually considered uniform, which is not the case for increasingly larger focal plane arrays (FPA). In this article we propose a intensity calibration scheme for CEA-LETIs LiDAR prototype in order to compensate the nonuniform illumination of the target and present the first enhanced results based on data acquired under laboratory conditions.

In recent years, graph embedding has become a significant technique to deal with feature extraction and dimension reduction problems. Under the linearization and kernelization, it provides a unified framework in machine learning and other pattern recognition tasks. Polarimetric synthetic aperture (PolSAR) as a typical multi-channel sensor can obtain more geometrical and geophysical information. How to combine those polarimetric scattering signals and target decomposition features and explore the spatial information between pixels become a new research direction to address PolSAR data. In this paper, we utilize the tensorial embedding methods to extract the intrinsic features from a redundant feature space for the PolSAR land cover classification. The effectiveness of the proposed methods is demonstrated using AIRSAR Flevoland data set.

Among direct and diffuse components that compose photosynthetically active radiation (PAR), diffuse component of sunlight is important to evaluate fraction absorbed PAR (FAPAR) of plants because diffuse flux penetrates more deeply than direct flux and the peak of photosynthetic photon flux density (PPFD) use efficiency occurs at low to medium PPFD. Shading distribution in leaf-scale aerial images of plants by low-altitude measurement via UAVs contains sunlight information as well as plant structure and leaf pigments. In this work, we investigate the relationship between statistical properties of leaf-scale images, share of diffuse flux (SDF) in global radiation and solar zenith angles (SZA). Higher-order statistics (HOSs) were calculated from ground-based close-range images of wheat leaves under various sunlight conditions. SDF were measured based on two field spectrometers. We confirmed that (1) images under clear sky can be distinguished from those under cloudy sky based on HOSs, and (2) it is possible to estimate SZA based on HOSs of leaf-scale images under clear sky.

This paper presents four dimensionality reduction methods, supervised mixtures of factor analyzers (SMFA), semi-supervised mixtures of factor analyzers (S2MFA), supervised deep mixtures of factor analyzers (SDMFA) and semi-supervised deep mixtures of factor analyzers (S2DMFA), for hyperspectral image (HSI) classification. The performance of SMFA, S2MFA, SDMFA, and S2DMFA dimensionality reduction methods for classification using real HSI is evaluated in this paper. Experimental results are compared to more conventional methods like probabilistic principal component analysis, factor analysis, mixtures of factor analyzers and deep mixtures of factor analyzers and it is shown that the proposed methods give better results.

This paper presents two dimensionality reduction methods, mixtures of factor analyzers (MFA) and deep mixtures of factor analyzers (DMFA), for classification of hyperspectral image (HSI). DMFA consists of two layers of MFA and can extract more information from HSI than MFA can. The performance of MFA and DMFA dimensionality reduction methods for classification using real HSI is evaluated in this paper. Experimental results are compared to conventional methods like probabilistic principal component analysis and factor analysis and it is shown that MFA and DMFA give better results.

A method based on spectral unmixing (SU) for snow cover estimation from a time series of optical images is proposed in this letter. Specifically, we have developed an endmember estimation procedure that exploits the temporal continuity of a scene. Consecutive dates are jointly processed for a more precise description of background materials improving the estimation of a fractional snow cover map. The proposed workflow relies on up to three consecutive acquisitions over the same area to extract an appropriate set of background spectra to be considered as endmembers. In greater details, multiple sets of endmembers are extracted from different images in the time series by a geometrical automated endmember extraction algorithm and the most relevant one is selected in terms of reconstruction error. Snow cover maps are then estimated by SU considering as endmembers the snow spectra coming from a spectral library and those associated with the background materials as estimated by the proposed procedure. The proposed technique is quantitatively validated considering Moderate-Resolution Imaging Spectroradiometer Terra data over the French Alps and Moroccan High Atlas image time series and comparing the estimated snow cover maps with high-resolution reference data. The experiment clearly demonstrates the effectiveness of the generated set of endmembers using three different approaches to abundance estimation.

Pansharpening refers to the problem of fusing a multispectral (MS) image and a panchromatic image in order to get an MS image at a finer spatial resolution than the one of the original MS image. Due to the lack of a reference image, the assessment of the quality of a pansharpened product is a challenging task. Typical solutions are related to the reduced resolution assessment by exploiting Wald's protocol or to indexes without reference. In this letter, an efficient approach based on the combination of the two above-mentioned solutions is proposed. The performance is assessed using real data acquired by sensors with very different features mounted on-board of the Pléiades, the WorldView-3, and the WorldView-4 satellite platforms.

Over the last years, several stationarity tests have been proposed. One of these methods uses time-frequency representations and stationarized replicas of the signal (known as surrogates) for testing wide-sense stationarity. In this letter, we propose a procedure to improve the original surrogate test. The proposed methodology can be seen as a guideline on how the surrogate test can be improved. We show mathematically that the modified test should exhibit improved classification performance. Numerical simulations on synthetic and real-world signals are carried out to evaluate the modified test against competing ones.

Scene classification of remote sensing images has drawn great attention because of its wide applications. In this paper, with the guidance of the human visual system (HVS), we explore the attention mechanism and propose a novel end-to-end attention recurrent convolutional network (ARCNet) for scene classification. It can learn to focus selectively on some key regions or locations and just process them at high-level features, thereby discarding the noncritical information and promoting the classification performance. The contributions of this paper are threefold. First, we design a novel recurrent attention structure to squeeze high-level semantic and spatial features into several simplex vectors for the reduction of learning parameters. Second, an end-to-end network named ARCNet is proposed to adaptively select a series of attention regions and then to generate powerful predictions by learning to process them sequentially. Third, we construct a new data set named OPTIMAL-31, which contains more categories than popular data sets and gives researchers an extra platform to validate their algorithms. The experimental results demonstrate that our model makes great promotion in comparison with the state-of-the-art approaches.

Image classification is considered to be one of the critical tasks in hyperspectral remote sensing image processing. Recently, a convolutional neural network (CNN) has established itself as a powerful model in classification by demonstrating excellent performances. The use of a graphical model such as a conditional random field (CRF) contributes further in capturing contextual information and thus improving the classification performance. In this paper, we propose a method to classify hyperspectral images by considering both spectral and spatial information via a combined framework consisting of CNN and CRF. We use multiple spectral band groups to learn deep features using CNN, and then formulate deep CRF with CNN-based unary and pairwise potential functions to effectively extract the semantic correlations between patches consisting of 3-D data cubes. Furthermore, we introduce a deep deconvolution network that improves the final classification performance. We also introduced a new data set and experimented our proposed method on it along with several widely adopted benchmark data sets to evaluate the effectiveness of our method. By comparing our results with those from several state-of-the-art models, we show the promising potential of our method.

With a large amount of open satellite multispectral (MS) imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global MS land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to MS ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace), by jointly considering subspace learning and supervised classification. By locally aligning the manifold structure of the two modalities, CoSpace linearly learns a shared latent subspace from hyperspectral-MS (HS-MS) correspondences. The MS out-of-samples can be then projected into the subspace, which are expected to take advantages of rich spectral information of the corresponding hyperspectral data used for learning, and thus leads to a better classification. Extensive experiments on two simulated HS-MS data sets (University of Houston and Chikusei), where HS-MS data sets have tradeoffs between coverage and spectral resolution, are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

Climate change and anthropogenic pressure are causing an indisputable decline in biodiversity; therefore, the need of environmental knowledge is important to develop the appropriate management plans. In this context, remote sensing and, specifically, hyperspectral imagery (HSI) can contribute to the generation of vegetation maps for ecosystem monitoring. To properly obtain such information and to address the mixed pixels inconvenience, the richness of the hyperspectral data allows the application of unmixing techniques. In this sense, a problem found by the traditional linear mixing model (LMM), a fully constrained least squared unmixing (FCLSU), is the lack of ability to account for spectral variability. This paper focuses on assessing the performance of different spectral unmixing models depending on the quality and quantity of endmembers. A complex mountainous ecosystem with high spectral changes was selected. Specifically, FCLSU and 3 approaches, which consider the spectral variability, were studied: scaled constrained least squares unmixing (SCLSU), Extended LMM (ELMM) and Robust ELMM (RELMM). The analysis includes two study cases: 1) robust endmembers and 2) nonrobust endmembers. Performances were computed using the reconstructed root-mean-square error (RMSE) and classification maps taking the abundances maps as inputs. It was demonstrated that advanced unmixing techniques are needed to address the spectral variability to get accurate abundances estimations. RELMM obtained excellent RMSE values and accurate classification maps with very little knowledge of the scene and minimum effort in the selection of endmembers, avoiding the curse of dimensionality problem found in HSI.

Spatiotemporal image fusion is considered as a promising way to provide Earth observations with both high spatial resolution and frequent coverage, and recently, learning-based solutions have been receiving broad attention. However, these algorithms treating spatiotemporal fusion as a single image super-resolution problem, generally suffers from the significant spatial information loss in coarse images, due to the large upscaling factors in real applications. To address this issue, in this paper, we exploit temporal information in fine image sequences and solve the spatiotemporal fusion problem with a two-stream convolutional neural network called StfNet. The novelty of this paper is twofold. First, considering the temporal dependence among image sequences, we incorporate the fine image acquired at the neighboring date to super-resolve the coarse image at the prediction date. In this way, our network predicts a fine image not only from the structural similarity between coarse and fine image pairs but also by exploiting abundant texture information in the available neighboring fine images. Second, instead of estimating each output fine image independently, we consider the temporal relations among time-series images and formulate a temporal constraint. This temporal constraint aiming to guarantee the uniqueness of the fusion result and encourages temporal consistent predictions in learning and thus leads to more realistic final results. We evaluate the performance of the StfNet using two actual data sets of Landsat-Moderate Resolution Imaging Spectroradiometer (MODIS) acquisitions, and both visual and quantitative evaluations demonstrate that our algorithm achieves state-of-the-art performance.

The use of cutting-edge geospatial technologies to monitor ecosystems and the development of tailored tools for assessing such natural areas is a fundamental task. In this context, the growing availability of hyperspectral (HS) imagery from satellite and aerial platforms can provide valuable information for the sustainable management of ecosystems. However, in some cases, the spectral richness provided by HS sensors is at the expense of spatial quality. To alleviate this inconvenience, which can be critical to monitor some heterogeneous and mixed natural areas, a number of HS sharpening techniques have been developed to increase the spatial resolution while trying to preserve the spectral content. This image processing field has attracted the interest of the scientific community, and many research studies have been conducted to assess the performance of different HS sharpening algorithms. In the last decade, however, many comparative studies rely upon simulated data. In this work, the challenging application of sharpening methods in real situations using multiplatform or multisensor data is also addressed. Thus, experiments with real data have been conducted, in addition to a thorough assessment of HS sharpening techniques using simulated imagery in scenarios with different spatial resolution ratios and registration errors. In particular, airborne and satellite HS imageries have been pansharpened with drone, orthophotos, and satellite high spatial resolution data evaluating 11 fusion algorithms. After a comprehensive analysis, considering different visual and quantitative quality indicators, the algorithm characteristics have been summarized and the methods with higher performance and robustness have been identified.

Semantic segmentation requires methods capable of learning high-level features while dealing with large volume of data. Toward such goal, convolutional networks can learn specific and adaptable features based on the data. However, these networks are not capable of processing a whole remote sensing image, given its huge size. To overcome such limitation, the image is processed using fixed size patches. The definition of the input patch size is usually performed empirically (evaluating several sizes) or imposed (by network constraint). Both strategies suffer from drawbacks and could not lead to the best patch size. To alleviate this problem, several works exploited multicontext information by combining networks or layers. This process increases the number of parameters, resulting in a more difficult model to train. In this paper, we propose a novel technique to perform semantic segmentation of remote sensing images that exploits a multicontext paradigm without increasing the number of parameters while defining, in training time, the best patch size. The main idea is to train a dilated network with distinct patch sizes, allowing it to capture multicontext characteristics from heterogeneous contexts. While processing these varying patches, the network provides a score for each patch size, helping in the definition of the best size for the current scenario. A systematic evaluation of the proposed algorithm is conducted using four high-resolution remote sensing data sets with very distinct properties. Our results show that the proposed algorithm provides improvements in pixelwise classification accuracy when compared to the state-of-the-art methods.

The combination of a multispectral (MS) image and a panchromatic (PAN) image, the so-called pansharpening, allows to produce very appealing images that are useful both for visual interpretation and for feature extraction. The state-of-the-art multiresolution analysis pansharpening algorithms are based on the extraction of spatial details from the PAN image through image filters matched with the MS sensors' modulation transfer function. However, this knowledge is often poor due to measurement inaccuracies and/or its aging. Thus, deconvolution algorithms have been proposed to overcome this limitation. In this paper, we propose a multiband filter estimation (FE) approach to improve the solutions in the literature. The main idea in this paper is to exploit a preliminary pansharpened image to estimate the spatial filter used for detail extraction associated with each spectral band. We demonstrate that the proposed method outperforms the state-of-the-art FE approaches by employing data sets acquired by the IKONOS, the Quickbird, and the WorldView-3 sensors.

With the rapid development of spaceborne imaging techniques, object detection in optical remote sensing imagery has drawn much attention in recent decades. While many advanced works have been developed with powerful learning algorithms, the incomplete feature representation still cannot meet the demand for effectively and efficiently handling image deformations, particularly objective scaling and rotation. To this end, we propose a novel object detection framework, called Optical Remote Sensing Imagery detector (ORSIm detector), integrating diverse channel features extraction, feature learning, fast image pyramid matching, and boosting strategy. An ORSIm detector adopts a novel spatial-frequency channel feature (SFCF) by jointly considering the rotation-invariant channel features constructed in the frequency domain and the original spatial channel features (e.g., color channel and gradient magnitude). Subsequently, we refine SFCF using learning-based strategy in order to obtain the high-level or semantically meaningful features. In the test phase, we achieve a fast and coarsely scaled channel computation by mathematically estimating a scaling factor in the image domain. Extensive experimental results conducted on the two different airborne data sets are performed to demonstrate the superiority and effectiveness in comparison with the previous state-of-the-art methods.

Hyperspectral images provide much more information than conventional imaging techniques, allowing a precise identification of the materials in the observed scene, but because of the limited spatial resolution, the observations are usually mixtures of the contributions of several materials. The spectral unmixing problem aims at recovering the spectra of the pure materials of the scene (endmembers), along with their proportions (abundances) in each pixel. In order to deal with the intra-class variability of the materials and the induced spectral variability of the endmembers, several spectra per material, constituting endmember bundles, can be considered. However, the usual abundance estimation techniques do not take advantage of the particular structure of these bundles, organized into groups of spectra. In this paper, we propose to use group sparsity by introducing mixed norms in the abundance estimation optimization problem. In particular, we propose a new penalty, which simultaneously enforces group and within-group sparsity, to the cost of being nonconvex. All the proposed penalties are compatible with the abundance sum-to-one constraint, which is not the case with traditional sparse regression. We show on simulated and real datasets that well-chosen penalties can significantly improve the unmixing performance compared to classical sparse regression techniques or to the naive bundle approach.

Hyperspectral imagery collected from airborne or satellite sources inevitably suffers from spectral variability, making it difficult for spectral unmixing to accurately estimate abundance maps. The classical unmixing model, the linear mixing model (LMM), generally fails to handle this sticky issue effectively. To this end, we propose a novel spectral mixture model, called the augmented linear mixing model (ALMM), to address spectral variability by applying a data-driven learning strategy in inverse problems of hyperspectral unmixing. The proposed approach models the main spectral variability (i.e., scaling factors) generated by variations in illumination or typography separately by means of the endmember dictionary. It then models other spectral variabilities caused by environmental conditions (e.g., local temperature and humidity, atmospheric effects) and instrumental configurations (e.g., sensor noise), as well as material nonlinear mixing effects, by introducing a spectral variability dictionary. To effectively run the data-driven learning strategy, we also propose a reasonable prior knowledge for the spectral variability dictionary, whose atoms are assumed to be low-coherent with spectral signatures of endmembers, which leads to a well-known low-coherence dictionary learning problem. Thus, a dictionary learning technique is embedded in the framework of spectral unmixing so that the algorithm can learn the spectral variability dictionary and estimate the abundance maps simultaneously. Extensive experiments on synthetic and real datasets are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

This paper presents a hypserspectral image (HSI) super-resolution method which fuses a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) to get high-resolution HSI (HR-HSI). The proposed method first extracts the nonlocal similar patches to form a nonlocal patch tensor (NPT). A novel tensor-tensor product (t-product) based tensor sparse representation is proposed to model the extracted NPTs. Through the tensor sparse representation, both the spectral and spatial similarities between the nonlocal similar patches are well preserved. Then, the relationship between the HR-HSI and LR-HSI is built using t-product which allows us to design a unified objective function to incorporate the nonlocal similarity, tensor dictionary learning, and tensor sparse coding together. Finally, Alternating Direction Method of Multipliers (ADMM) is used to solve the optimization problem. Experimental results on three data sets and one real data set demonstrate that the proposed method substantially outperforms the existing state-of-the-art HSI super-resolution methods.

Hyperspectral image unmixing is a source separation problem aiming at recovering the spectra of the pure materials of the observed scene (called endmembers), as well as their relative proportions in each pixel of the image (called abundances). The variability of the materials has recently received a lot of attention in the community. In particular, a consequent number of models and algorithms have been proposed to estimate pixel-wise endmembers to account for their variability. These algorithms often rely on classical endmem-ber extraction algorithms to provide reference spectra. In difficult scenarios with shadows and significant variability these algorithms may fail. In this paper, we address this issue in the Extended Linear Mixing Model framework by considering that an endmember is a direction in the feature space, rather than a single point. Under this paradigm, we show that using k- means clustering with the cosine similarity outperforms geometric endmember extraction algorithms. We also design an algorithm to refine the estimation of the endmember directions, and to account for both illumination and intrinsic variability effects. We show the potential of the proposed algorithm on a synthetic dataset using real world spectra with variability, and a challenging real dataset of a natural scene.

Hyperspectral imagery (HSI) has become an important tool in ecosystem conservation due to its capability to perform accurate spectral unmixing, for vegetation mapping and ecosystem monitoring. An issue to be solved is the spectral variability of endmembers that can be induced by sensor noise and topographic changes. This spectral variability is considered by the Extended Linear Mixing Model (ELMM), which is applied to a mountainous ecosystem with high spectral variability and radiometric changes in each swath. The results obtained are very satisfactory, achieving reasonable abundance estimations and accurate characterization of the variability within the scene. ELMM allows studying the features of each pixel, including additional information about the characterization of the mixed pixels, in HSI by taking spectral variability into account. Moreover, it is observed that ELMM is robust to the absence of pure pixels as well as to noise.

Ecosystem management and monitoring are essential to preserve natural resources. Hyperspectral imagery (HSI) is a useful tool to obtain accurate classification maps, providing significant level of detail. Thus, traditional and novel methodologies based on pixel and object classification approaches are compared and evaluated in a homogeneous and mixed vulnerable ecosystem. Considering the challenging ecosystem, all classifications successfully resulted in high OA (higher than 82%), showing that HSI is very useful providing accurate vegetation maps to evaluate and monitor the ecosystems in a faster and economic way.

Multispectral pan-sharpening plays an important role in providing earth observation with both high-spatial and high-spectral resolutions, and recently pan-sharpening with machine learning has been attracting broad interest. However, these algorithms minimizing the pixel-wise mean squared error, generally suffer from over-smoothed results that lack of high-frequency details in both spatial and spectral dimensions. In this paper, we propose to tackle this problem by shifting the learning loss from pixel-wise error to a higher-level feature loss. The new loss function, formulated by spatial structure similarity and spectral angle mapping, pushes the model to generate results that have similar feature representations with ground truth, rather than match with pixel-wise accuracy. Consequently, more realistic fusion results can be produced. Visual and quantitative analysis both demonstrate that our approach achieves better performance in comparison with state-of-the-art algorithms. Furthermore, experiments on high-level remote sensing task further confirm the superiority of the proposed method in real applications.

There is an inevitable trade-off between spatial and spectral resolutions in optical remote sensing images. A number of data fusion techniques of multimodal images with different spatial and spectral characteristics have been developed to generate optical images with both spatial and spectral high resolution. Although some of the techniques take the spectral and spatial blurring process into account, there is no method that attempts to retrieve an optical image with both spatial and spectral high resolution, a spectral blurring filter and a spectral response simultaneously. In this paper, we propose a new framework of spatial resolution enhancement by a fusion of multiple optical images with different characteristics based on tensor decomposition. An optical image with both spatial and spectral high resolution, together with a spatial blurring filter and a spectral response, is generated via canonical polyadic (CP) decomposition of a set of tensors. Experimental results featured that relatively reasonable results were obtained by regularization based on nonnegativity and coupling.

Pansharpening regards the fusion of a high spatial resolution but low spectral resolution (panchromatic) image with a high spectral resolution but low spatial resolution (multispectral) image. The estimation, at reduced resolution, of injection coefficients through regression is a widespread and powerful approach. In this work, the problem of the estimation of the injection coefficients at full resolution for regression-based pansharpening approaches is studied. Multiple approaches (based on guess images or an iterative method) are proposed. These are assessed at reduced resolution by exploiting a real dataset acquired by the IKONOS sensor. The quantitative results clearly demonstrate the superiority of the proposed iterative method.

Gas plume detection is a challenging task in the field of remote sensing. Due to sensor development, it is now possible to detect and track chemical gas plumes with hyperspectral video sequences (HVS). For the purpose of high detection accuracy, it is important to explore spectral characteristic, and take full advantages of spatial continuity and temporal consistency in HVS. However, the high computational complexity and the large amount of data limit its application in time-critical scenarios. In this paper, we propose a GPU parallel implementation of gas plume detection in HVS, which properly exploits shared memory and intrinsic concurrency of the CUDA blocks, as well as parallel workload assignment and resource allocation. The experimental results demonstrate the proposed GPU parallel method has a considerable acceleration factor while retaining the same detection accuracy compared with the serial and multicore version.

Abstract This work addresses the problem of denoising image sequences through an approach that makes use of spatio‐temporal cellular automata‐based filtering. The algorithm is called st‐CAF and one of its key aspects is that the resulting cellular automata contemplate a spatio‐temporal neighbourhood when processing each pixel of the sequence. Additionally, the way the rule sets for the cellular automata are obtained, through evolutionary means, is also relevant, as it allows a good adaptation to any type of image and noise through the appropriate training set. This results in a great advantage over more traditional single frame denoising techniques presented in the literature or even over their adaptation to sequences. A fact that is made relevant in this paper through the application of the algorithm to different types of noisy images and its comparison to other techniques.

Multiple types of features, e.g., spectral, filtering, texture, and shape features, are helpful for hyperspectral image (HSI) classification tasks. Combining multiple features can describe the characteristics of pixels from different perspectives, and always results in better classification performance. Recently, multifeature combination learning has been widely employed to the multitask-learning-based representation-based model to obtain a multifeature representation vector. However, the linear sparse representation-based classifier (SRC) cannot handle the HSI with highly nonlinear distribution, and kernel sparse representation-based classifier (KSRC) can remedy the drawback of linear SRC. By adopting nonlinear mapping, the samples in kernel space are often of high or even infinite dimensionality. In this paper, we integrate kernel principal component analysis into multifeature-based KSRC and propose a novel multiple feature kernel sparse representation-based classifier (namely, MFKSRC) for hyperspectral imagery. More specifically, spatial features, Gabor textures, local binary patterns, and difference morphological profiles are adopted and then each kind of feature is transformed nonlinearly into a new low-dimensional kernel space. The proposed framework can handle data with nonlinear distribution and add a dimensionality reduction stage in kernel space before optimizing the corresponding cost function. Experimental results on different HSIs demonstrate that the proposed MFKSRC algorithm outperforms the state-of-the-art classifiers.

Pansharpening usually refers to the fusion of a high spatial resolution panchromatic (PAN) image with a higher spectral resolution but coarser spatial resolution multispectral (MS) image. Owing to the wide applicability of related products, the literature has been populated by many papers proposing several approaches and studies about this issue. Many solutions require a preliminary spectral matching phase wherein the PAN image is matched with the MS bands. In this paper, we propose and properly justify a new approach for performing this step, demonstrating that it yields state-of-the-art performance. The comparison with existing spectral matching procedures is performed by employing four data sets, concerning different kinds of landscapes, acquired by the Pléiades, WorldView-2, and GeoEye-1 sensors.

Pansharpening regards the fusion of a high-spatial resolution panchromatic image with a low-spatial resolution multispectral image. One of the most debated topics about pansharpening is related to the quality assessment of fused products. Two main assessment procedures are usually exploited in the literature: the reduced resolution validation and the full-resolution (FR) validation. The former has the advantage to be accurate, but the hypothesis of invariance among scales has to be assumed. The latter overcomes this limitation but paying it with a lower accuracy. In this paper, we will focus on the FR assessment proposing an approach for estimating an overall quality index at FR by using multiscale FR measurements. The problem is recast into the sequential Bayesian framework exploiting a Kalman filter to find its solution. The proposed procedure for quality evaluation has been tested on four real data sets acquired by the Pléiades, the GeoEye-1, the WorldView-3, and the WorldView-4 sensors assessing the quality of 19 pansharpened methods. The proposed approach has demonstrated its superiority with respect to the benchmark consisting of state-of-the-art quality assessment procedures.

Hyperspectral video sequences (HVSs) are well suited for gas plume detection (GPD). The high spectral resolution allows the detection of chemical clouds even when they are optically thin. Processing this new type of video sequences is challenging and requires advanced image and video analysis algorithms. In this paper, we propose a novel method for GPD recorded in HVSs. Based on the assumption that the background is stationary and the gas plume is moving, the proposed method separates the background from the gas plume via a low-rank and sparse decomposition. Furthermore, taking into consideration that the gas plume is continuous in both spatial and temporal dimensions, we include total variation regularization in the constrained minimization problem, which we solve using the augmented Lagrangian multiplier method. After applying the above process to each extracted feature, a novel fusion strategy is proposed to combine the information into a final detection result. Experimental results using real data sets indicate that the proposed method achieves very promising GPD performance.

Anomaly detection plays an important role in remotely sensed hyperspectral image (HSI) processing. Recently, compressive sensing technology has been widely used in hyperspectral imaging. However, the reconstruction from compressive HSI and detection are commonly completed independently, which will reduce the processing's efficiency and accuracy. In this paper, we propose a framework for hyperspectral compressive sensing with anomaly detection which reconstruct the HSI and detect the anomalies simultaneously. In the proposed method, the HSI is composed of the background and anomaly parts in the tensor robust principal component analysis model. To characterize the low-dimensional structure of the background, a novel tensor nuclear norm is used to constrain the background tensor. As the anomaly part is formed by a few anomalous spectra, the anomaly part is assumed to be a tuber-wise sparse tensor. In addition, to enhance the separation of the background and anomaly, we minimize the sum of Mahalanobis distance of the background pixels. Experiments on four HSIs demonstrate that the proposed method outperforms several state-of-the-art methods on both reconstruction and anomaly detection accuracies.

Pansharpening is an important application in remote sensing image processing. It can increase the spatial-resolution of a multispectral image by fusing it with a high spatial-resolution panchromatic image in the same scene, which brings great favor for subsequent processing such as recognition, detection, etc. In this paper, we propose a continuous modeling and sparse optimization based method for the fusion of a panchromatic image and a multispectral image. The proposed model is mainly based on reproducing kernel Hilbert space (RKHS) and approximated Heaviside function (AHF). In addition, we also propose a Toeplitz sparse term for representing the correlation of adjacent bands. The model is convex and solved by the alternating direction method of multipliers which guarantees the convergence of the proposed method. Extensive experiments on many real datasets collected by different sensors demonstrate the effectiveness of the proposed technique as compared with several state-of-the-art pansharpening approaches.

Pansharpening is usually related to the fusion of a high spatial resolution but low spectral resolution (panchromatic) image with a high spectral resolution but low spatial resolution (multispectral) image. The calculation of injection coefficients through regression is a very popular and powerful approach. These coefficients are usually estimated at reduced resolution. In this paper, the estimation of the injection coefficients at full resolution for regression-based pansharpening approaches is proposed. To this aim, an iterative algorithm is proposed and studied. Its convergence, whatever the initial guess, is demonstrated in all the practical cases and the reached asymptotic value is analytically calculated. The performance is assessed both at reduced resolution and at full resolution on four data sets acquired by the IKONOS sensor and the WorldView-3 sensor. The proposed full scale approach always shows the best performance with respect to the benchmark consisting of state-of-the-art pansharpening methods.

Local Spectral Unmixing (LSU) methods perform the unmixing of hyperspectral data locally in regions of the image. The endmembers and their abundances in each pixel are extracted region-wise, instead of globally to mitigate spectral variability effects, which are less severe locally. However, it requires the local estimation of the number of endmembers to use. Algorithms for intrinsic dimensionality (ID) estimation tend to overestimate the local ID, especially in small regions. The ID only provides an upper bound of the application and scale dependent number of endmembers, which leads to extract irrelevant signatures as local endmembers, associated with meaningless local abundances. We propose a method to select in each region the best subset of the locally extracted endmembers. Collaborative sparsity is used to detect spurious endmembers in each region and only keep the most influent ones. We compute an algorithmic regularization path for this problem, giving access to the sequence of successive active sets of endmembers when the regularization parameter is increased. Finally, we select the optimal set in the sense of the Bayesian Information Criterion (BIC), favoring models with a high likelihood, while penalizing those with too many endmembers. Results on real data show the interest of the proposed approach.

Most of the tests proposed in the literature to verify if a given random multivariate dataset fits a spherical or elliptical distribution are designed for real valued data and rely on the estimation of high order moment matrices. Recently, a test that considers complex random vectors, derived based on the Schott spherical symmetry test was proposed aiming in a more proper analysis of PolSAR data. Results showed its effectiveness in discriminating data that fits or not the complex spherically invariant random vector model (product model), inherent to high resolution heterogeneous PolSAR systems. Within this context, this paper further extends the assessment of the referred test efficiency, verifying its performance under different stochastic model assumptions and comparing the results with the ones achieved when the Schott test derived for real random vectors is employed.

Inpainting in hyperspectral imagery is a challenging research area and several methods have been recently developed to deal with this kind of data. In this paper we address missing data restoration via a convex optimization technique with regularization term based on Collaborative Total Variation (CTV). In particular we evaluate the effectiveness of several instances of CTV in conjunction with different dimensionality reduction algorithms.

In this paper, we propose a continuous modeling and sparse optimization based method for the fusion of a panchromatic (PAN) image and a multispectral (MS) image. The proposed model is mainly based on reproducing kernel Hilbert space (RKHS) and approximated Heaviside function (AHF). In addition, we also design an iterative strategy to recover more image details. The final model is a convex one and solved by the designed alternating direction method of multipliers (ADMM) which guarantees the convergence of the proposed method. Experimental results on two real datasets corresponding to different sensors and different resolutions demonstrate the effectiveness of the proposed approach as compared with several state-of-the-art pansharpening approaches.

This paper presents a novel spectral mixture model to address spectral variability in inverse problems of hyperspectral unmixing. Based on the linear mixture model (LMM), our model introduces a spectral variability dictionary to account for any residuals that cannot be explained by the LMM. Atoms in the dictionary are assumed to be low-coherent with spectral signatures of endmembers. A dictionary learning technique is proposed to learn the spectral variability dictionary while solving unmixing problems simultaneously. Experimental results on synthetic and real datasets demonstrate that the performance of the proposed method is superior to state-of-the-art methods.

In the past few years, graph-based methods have proven to be a useful tool in a wide variety of energy minimization problems [1]. In this paper, we propose a graph-based algorithm for feature extraction and segmentation of multimodal images. By defining a notion of similarity that integrates information from each modality, we merge the different sources at the data level. The graph Laplacian then allows us to perform feature extraction and segmentation on the fused dataset. We apply this method in a practical example, namely the segmentation of optical and lidar images. The results obtained confirm the potential of the proposed method.

Spectral unmixing is a central problem in hyperspectral imagery. It is usually assuming a linear mixture model. Solving this inverse problem, however, can be seriously impacted by a wrong estimation of the number of endmembers, a bad estimation of the endmembers themselves, the spectral variability of the endmembers or the presence of nonlinearities. These problems can result in a too large number of retained endmembers. We propose to tackle this problem by introducing a new formulation for robust linear unmixing enhancing sparsity. With a single tuning parameter the optimization leads to a range of behaviors: from the standard linear model (low sparsity) to a hard classification (maximal sparsity : only one endmember is retained per pixel). We solve the proposed new functional using a computationally efficient proximal primal dual method. The experimental study, including both realistic simulated data and real data demonstrates the versatility of the proposed approach.

Variational methods are widely used in image processing for problems ranging from denoising to data fusion. In this paper we focus on a recent regularization method, called Collaborative Total Variation, applied to the hyperspectral pansharpening, which deals with the fusion of low resolution hyperspectral and high resolution panchromatic images. The effectiveness of this novel approach is evaluated for different Collaborative Norms and the assessment is performed on the Pavia University dataset.

We propose to use the temporal coherence of a time series to extract using Vertex Component Analysis (VCA) the suitable set of endmembers for each scene. The reconstruction error computed on the two previous scenes for each date is used to constrain the selection of the set of endmembers produced by VCA. Snow cover estimation is considered as application. We tested different approaches for abundance estimation (FCLSU, SUnSAL, ELMM) over the French Alps from Moderate Resolution Imaging Spectroradiometer (MODIS) images. Results shows a decrease of the false positive rate with the proposed approach.

Cloude and Pottier H/α feature space [1] is one of the most employed methods for unsupervised PolSAR data classification based on Incoherent Target Decomposition. The association of the coherence matrix eigenvectors to the most dominant scatters in the analysed pixel introduces unfeasible regions in the H/α plane. The Independent Component Analysis provides promising new information to better interpret non-Gaussian heterogeneous clutter in the frame of polarimetric incoherent target decompositions. Not constrained to any orthogonality between the estimated scattering mechanisms that compose the clutter under analysis, ICA does not introduce any unfeasible region in the H/α plane, increasing the range of possible natural phenomenons depicted in the aforementioned feature space.

This work describes a novel spatio-temporal cellular automata-based filtering algorithm (st-CAF) intended for performing image sequence denoising processes. The approach presents several advantages over more traditional single frame denoising techniques presented in the literature or even over their adaptation to sequences. Especially the fact that the cellular automaton used is able to contemplate information concerning the type of noise through the use of specific sequences to tune the algorithm, as well as temporal information by means of a spatio-temporal neighborhood when processing each pixel of the sequence. These two elements lead to significant improvements in the results with respect to simple spatial or temporal sets of neighbors.

Classical pansharpening algorithms constitute a class of image fusion methods that have been widely investigated in the literature. They have been developed for combining a single- and a multichannel image (panchromatic (PAN) and multispectral (MS), respectively), but can be adapted to the sharpening of hyperspectral (HS) data, both through companion PAN and MS images. We focus in this letter on the HS/MS fusion, showing that the assignation of the MS channel to each HS band is a key step, and investigate several alternatives to make this choice. The assignment algorithms are tested in conjunction with both component substitution and multiresolution analysis pansharpening methods and assessed on images acquired by the Hyperion and ALI sensors. The numerical evaluation shows that the best results can be obtained by optimizing the spectral angle mapper metric confirming that classical methods represent a reliable basis for the development of novel sharpening algorithms.

Hyperspectral image unmixing is a source separation problem whose goal is to identify the signatures of the materials present in the imaged scene (called endmembers), and to estimate their proportions (called abundances) in each pixel. Usually, the contributions of each material are assumed to be perfectly represented by a single spectral signature and to add up in a linear way. However, the main two limitations of this model have been identified as nonlinear mixing phenomena and spectral variability, i.e., the intraclass variability of the materials. The former limitation has been addressed by designing nonlinear mixture models, whereas the second can be dealt with by using (usually linear) space varying models. The typical example is a linear mixing model where the sources can vary from one pixel to the other. In this letter, we show that a recent variability model can also estimate the abundances of nonlinear mixtures to some extent. We make the theoretical connection between nonlinear models and this variability model, and confirm it with experiments on nonlinearly generated synthetic datasets.

With the rapid development of spectral imaging techniques, classification of hyperspectral images (HSIs) has attracted great attention in various applications such as land survey and resource monitoring in the field of remote sensing. A key challenge in HSI classification is how to explore effective approaches to fully use the spatial-spectral information provided by the data cube. Multiple kernel learning (MKL) has been successfully applied to HSI classification due to its capacity to handle heterogeneous fusion of both spectral and spatial features. This approach can generate an adaptive kernel as an optimally weighted sum of a few fixed kernels to model a nonlinear data structure. In this way, the difficulty of kernel selection and the limitation of a fixed kernel can be alleviated. Various MKL algorithms have been developed in recent years, such as the general MKL, the subspace MKL, the nonlinear MKL, the sparse MKL, and the ensemble MKL. The goal of this paper is to provide a systematic review of MKL methods, which have been applied to HSI classification. We also analyze and evaluate different MKL algorithms and their respective characteristics in different cases of HSI classification cases. Finally, we discuss the future direction and trends of research in this area.

With the development of hyperspectral sensors, nowadays, we can easily acquire large amount of hyperspectral images (HSIs) with very high spatial resolution, which has led to a better identification of relatively small structures. Owing to the high spatial resolution, there are much less mixed pixels in the HSIs, and the boundaries between these categories are much clearer. However, the high spatial resolution also leads to complex and fine geometrical structures and high inner-class variability, which make the classification results very “noisy.” In this paper, we propose a multimorphological superpixel (MMSP) method to extract the spectral and spatial features and address the aforementioned problems. To reduce the difference within the same class and obtain multilevel spatial information, morphological features (multistructuring element extended morphological profile or multiattribute filter extended multi-attribute profiles) are first obtained from the original HSI. After that, simple linear iterative clustering segmentation method is performed on each morphological feature to acquire the MMSPs. Then, uniformity constraint is used to merge the MMSPs belonging to the same class which can avoid introducing the information from different classes and acquire spatial structures at object level. Subsequently, mean filtering is utilized to extract the spatial features within and among MMSPs. At last, base kernels are obtained from the spatial features and original HSI, and several multiple kernel learning methods are used to obtain the optimal kernel to incorporate into the support vector machine. Experiments conducted on three widely used real HSIs and compared with several well-known methods demonstrate the effectiveness of the proposed model.

The Cloude and Pottier H/α feature space is one of the most employed methods for unsupervised polarimetric synthetic aperture radar (PolSAR) data classification based on incoherent target decomposition (ICTD). The method can be split in two stages: the retrieval of the canonical scattering mechanisms present in an image cell and their parameterization. The association of the coherence matrix eigenvectors to the most dominant scattering mechanisms in the analyzed pixel introduces unfeasible regions in the H/α plane. This constraint can compromise the performance of detection, classification, and geophysical parameter inversion algorithms that are based on the investigation of this feature space. The independent component analysis (ICA), recently proposed as an alternative to eigenvector decomposition, provides promising new information to better interpret non-Gaussian heterogeneous clutter (inherent to highresolution SAR systems) in the frame of polarimetric ICTDs. Not constrained to any orthogonality between the estimated scattering mechanisms that compose the clutter under analysis, ICA does not introduce any unfeasible region in the H/α plane, increasing the range of possible natural phenomena depicted in the aforementioned feature space. This paper addresses the potential of the new information provided by the ICA as an ICTD method with respect to Cloude and Pottier H/α feature space. A PolSAR data set acquired in October 2006 by the E-SAR system over the upper part of the Tacul glacier from the Chamonix Mont Blanc test site, France, and a RAMSES X-band image acquired over Brétigny, France, are taken into consideration to investigate the characteristics of pixels that may fall outside the feasible regions in the H/α plane that arise when the eigenvector approach is employed.

Pansharpened images are widely used synthetic representations of the Earth surface characterized by both a high spatial resolution and a high spectral diversity. They are usually generated by extracting spatial details from a high-resolution PANchromatic image and by injecting them into a low spatial resolution multispectral image. The details injection is performed through injection coefficients, whose values can be either uniform for the whole image (global methods) or spatially variant (context-adaptive (CA) approaches). In this paper, we propose a CA approach in which the injection coefficients are estimated over image segments achieved through a binary partition tree segmentation algorithm. The approach is applied to two credited pansharpening algorithms based on the Gram-Schmidt orthogonalization procedure and the generalized Laplacian pyramid technique. The performance assessment is performed using two different data sets acquired by the QuickBird and the WorldView-3 satellites. The validation procedure, both at full and at reduced resolution, shows the suitability of the proposed approach, which reaches a good tradeoff between accuracy and computational burden.

It is now possible to collect hyperspectral video sequences at a near real-time frame rate. The wealth of spectral, spatial, and temporal information of those sequences is appealing for various applications, but classical video processing techniques must be adapted to handle the high dimensionality and huge size of the data to process. In this paper, we introduce a novel method based on the hierarchical analysis of hyperspectral video sequences to perform object tracking. This latter operation is tackled as a sequential object detection process, conducted on the hierarchical representation of the hyperspectral video frames. We apply the proposed methodology to the chemical gas plume tracking scenario and compare its performances with state-of-the-art methods, for two real hyperspectral video sequences, and show that the proposed approach performs at least equally well.

We apply social ℓ-norms for the first time to the problem of hyperspectral unmixing while modeling spectral variability. These norms are built with inter-group penalties which are combined in a global intra-group penalization that can enforce selection of entire endmember bundles; this results in the selection of a few representative materials even in the presence of large endmembers bundles capturing each material's variability. We demonstrate improvements quantitatively on synthetic data and qualitatively on real data for three cases of social norms: group, elitist, and a fractional social norm, respectively. We find that the greatest improvements arise from using either the group or fractional flavor.

Thanks to the fast development of sensors, it is now possible to acquire sequences of hyperspectral images. Those hyperspectral video sequences (HVS) are particularly suited for the detection and tracking of chemical gas plumes. In this paper, we present a novel gas plume detection method. It is based on the decomposition of the sequence into a low-rank and a sparse term, corresponding to the background and the plume, respectively, and incorporating temporal consistency. To introduce spatial continuity, a post processing is added using the Total Variation (TV) regularized model. Experimental results on real hyperspectral video sequences validate the effectiveness of the proposed method.

Land cover classification is a task that requires methods capable of learning high-level features while dealing with high volume of data. Overcoming these challenges, Convolutional Networks (ConvNets) can learn specific and adaptable features depending on the data while, at the same time, learn classifiers. In this work, we propose a novel technique to automatically perform pixel-wise land cover classification. To the best of our knowledge, there is no other work in the literature that perform pixel-wise semantic segmentation based on data-driven feature descriptors for high-resolution remote sensing images. The main idea is to exploit the power of ConvNet feature representations to learn how to semantically segment remote sensing images. First, our method learns each label in a pixel-wise manner by taking into account the spatial context of each pixel. In a predicting phase, the probability of a pixel belonging to a class is also estimated according to its spatial context and the learned patterns. We conducted a systematic evaluation of the proposed algorithm using two remote sensing datasets with very distinct properties. Our results show that the proposed algorithm provides improvements when compared to traditional and state-of-the-art methods that ranges from 5 to 15% in terms of accuracy.

Recent advances in airborne light detection and ranging (LiDAR) technology allow us to rapid measure the topographical information over large areas. LiDAR remote sensed data has been widely used in many applications, e.g. forest management, urban planning, disaster predictions, etc. However, extracting useful information from LiDAR data remains challenging, especially in the urban remote sensing, where many objects have the same elevation and are connected, such as road and parking lots, trees and buildings. In this work, we present a new method to extract geometric and textural information from LiDAR data by using attribute filters with partial reconstruction. The proposed method can separate the connected objects and better model the geometric and textural information than traditional connected filters (e.g. attribute filters). Experimental results on LiDAR data from the 2013 IEEE GRSS Data Fusion Contest demonstrate effectiveness of the proposed method. Compared to the methods using original LiDAR data or attribute profiles computed by traditional attribute filters, with the proposed method, overall classification accuracies were improved by 35% and 12%, respectively.

Accurate representations of the Earth surface in both spatial and spectral domains are highly desirable in many applications using remotely sensed data. An effective solution is achieved by combining hyperspectral data, which are characterized by a high spectral diversity, with high spatial resolution images, collected by multispectral or panchromatic sensors. In this work, we compare the outcomes provided by fusing single-platform or multi-platform data. We demonstrate that the optimal choice depends on the target spatial resolution to be achieved. To this aim, real images collected by the Hyperion sensor are combined with data acquired by the ALI sensor or the QuickBird sensor assessing the fused outcomes at reduced resolution.

Morphological attribute profiles (APs) are discriminant features in the spectral–spatial classification of hyperspectral data. However, the optimal range of parameters in each filter is always a challenging yet important task, since an unsuitable range of parameters likely leads to inferior results. In order to alleviate this problem, we propose an ensemble method, which integrates multiple classification results based on a series of APs. The APs are obtained by using different filters with thresholds that are randomly selected from an arbitrarily defined range of parameters. Experimental results conducted on two hyperspectral images demonstrate the robustness and effectiveness of the proposed method.

In this letter, a novel subpixel sharpening for soft-then-hard subpixel mapping (SPM) is proposed. First, the fractional images for each class are, respectively, derived by spectral unmixing followed by spatial interpolation and by spectral interpolation followed by spectral unmixing. Bilinear and bicubic interpolation is used as the spatial and spectral interpolation methods. The fractional images for each class are then integrated together using the appropriate weighting parameter. Finally, the integrated finer fractional images are used to allocate hard class labels to subpixels. The proposed method is fast and does not need any prior spatial structure information. Experiments on two actual hyperspectral images show that the proposed method produces higher accuracy results than the existing algorithms. Moreover, both the spatial and spectral information is fully utilized to improve the accuracy of the SPM results.

In this letter, we propose a new version of the rotation forest (RoF) method for the pixelwise classification of hyperspectral images. RoF, which is an ensemble of decision tree classifiers, uses random feature selection and data transformation techniques (i.e., principal component analysis) to improve both the accuracy of base classifiers and the diversity within the ensemble. Traditional RoF performs data transformation on the training samples of each subset. In order to further improve the performance of RoF, the data transformation is separately performed on each class, extracting sets of transformation matrices that are strictly dependent on the training samples of each single class. The approach, namely, class-separation-based RoF (RoFCS), is experimentally investigated on a hyperspectral image collected by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor. Experimental results demonstrate that the proposed methodology achieves excellent performances, in comparison with random forest and RoF classifiers.

The intrinsic dimensionality (ID) of multivariate data is a very important concept in spectral unmixing of hyperspectral images. A good estimation of the ID is crucial for a correct retrieval of the number of endmembers (the spectral signatures of macroscopic materials) in the image, for dimensionality reduction or for subspace learning, among others. Recently, some approaches to perform spectral unmixing and superresolution locally have been proposed, which require a local estimation of the number of endmembers to use. However, the role of ID in local regions of hyperspectral images has not been properly addressed. Some important issues when dealing with small regions of hyperspectral data can seriously affect the performance of conventional hyperspectral ID estimators. We show that three factors mainly affect local ID estimation: the number of pixels in the local regions, which has to be high enough for the estimations to be relevant, the number of hyperspectral bands which complicates the estimations if the ambient space has a high dimensionality, and the noise, which can be misinterpreted as a signal when its power is important. Here, we review the hyperspectral ID estimators on the literature for local ID estimation, we show how they behave in a local setting on synthetic and real data sets, and we provide some guidelines to make proper use of these estimators in local approaches.

In this paper, we propose a novel multiple kernel learning (MKL) framework to incorporate both spectral and spatial features for hyperspectral image classification, which is called multiple-structure-element nonlinear MKL (MultiSE-NMKL). In the proposed framework, multiple structure elements (MultiSEs) are employed to generate extended morphological profiles (EMPs) to present spatial-spectral information. In order to better mine interscale and interstructure similarity among EMPs, a nonlinear MKL (NMKL) is introduced to learn an optimal combined kernel from the predefined linear base kernels. We integrate this NMKL with support vector machines (SVMs) and reduce the min-max problem to a simple minimization problem. The optimal weight for each kernel matrix is then solved by a projection-based gradient descent algorithm. The advantages of using nonlinear combination of base kernels and multiSE-based EMP are that similarity information generated from the nonlinear interaction of different kernels is fully exploited, and the discriminability of the classes of interest is deeply enhanced. Experiments are conducted on three real hyperspectral data sets. The experimental results show that the proposed method achieves better performance for hyperspectral image classification, compared with several state-of-the-art algorithms. The MultiSE EMPs can provide much higher classification accuracy than using a single-SE EMP.

Extended attribute profiles (EAPs) have been widely used for the classification of high-resolution hyperspectral images. EAPs are obtained by computing a sequence of attribute operators. Attribute filters (AFs) are connected operators, so they can modify an image by only merging its flat zones. These filters are effective when dealing with very high resolution images since they preserve the geometrical characteristics of the regions that are not removed from the image. However, AFs, being connected filters, suffer the problem of “leakage” (i.e., regions related to different structures in the image that happen to be connected by spurious links will be considered as a single object). Objects expected to disappear at a certain threshold remain present when they are connected with other objects in the image. The attributes of small objects will be mixed with their larger connected objects. In this paper, we propose a novel framework for morphological AFs with partial reconstruction and extend it to the classification of high-resolution hyperspectral images. The ultimate goal of the proposed framework is to be able to extract spatial features which better model the attributes of different objects in the remote sensed imagery, which enables better performances on classification. An important characteristic of the presented approach is that it is very robust to the ranges of rescaled principal components, as well as the selection of attribute values. Our experimental results, conducted using a variety of hyperspectral images, indicate that the proposed framework for AFs with partial reconstruction provides state-of-the-art classification results. Compared to the methods using only single EAP and stacking all EAPs computed by existing attribute opening and closing together, the proposed framework benefits significant improvements in overall classification accuracy.

In recent years, many studies on hyperspectral image classification have shown that using multiple features can effectively improve the classification accuracy. As a very powerful means of learning, multiple kernel learning (MKL) can conveniently be embedded in a variety of characteristics. This paper proposes a class-specific sparse MKL (CS-SMKL) framework to improve the capability of hyperspectral image classification. In terms of the features, extended multiattribute profiles are adopted because it can effectively represent the spatial and spectral information of hyperspectral images. CS-SMKL classifies the hyperspectral images, simultaneously learns class-specific significant features, and selects class-specific weights. Using an $L_{1}$-norm constraint (i.e., group lasso) as the regularizer, we can enforce the sparsity at the group/feature level and automatically learn a compact feature set for the classification of any two classes. More precisely, our CS-SMKL determines the associated weights of optimal base kernels for any two classes and results in improved classification performances. The advantage of the proposed method is that only the features useful for the classification of any two classes can be retained, which leads to greatly enhanced discriminability. Experiments are conducted on three hyperspectral data sets. The experimental results show that the proposed method achieves better performances for hyperspectral image classification compared with several state-of-the-art algorithms, and the results confirm the capability of the method in selecting the useful features.

The multiplicative model, expressed as a product between the square root of a scalar positive quantity (texture) and the description of an equivalent homogeneous surface (speckle), is one of the most appropriate and disseminated models used to describe high-resolution polarimetric synthetic aperture radar (PolSAR) clutter. Generally, the texture is assumed polarization independent, which causes PolSAR data to present a spherical symmetry property, allowing for the usage of most of the algorithms present in the literature. Nevertheless, the existence of polarization-dependent clutter has also been reported, for which specific algorithms need to be derived. Therefore, it becomes clear that the first step in SAR data analysis should be the validation of the model employed. Within this context, this paper presents a new methodological framework to assess the conformity of multivariate high-resolution SAR data with respect to the product model in terms of asymptotic statistics. More precisely, spherical symmetry is investigated by applying statistical hypothesis testing on the structure of the quadricovariance matrix. Simulated data, data from the P-band airborne data set acquired by the Office National d'Études et de Recherches Aérospatiales (ONERA) over the French Guiana in 2009 in the frame of the European Space Agency campaign TropiSAR and a RAMSES X-band image acquired over Brétigny, France, are taken into consideration to investigate the performance of the derived test. The detection results are qualitatively and quantitatively analyzed, and some important conclusions are drawn regarding the methodology employed in analyzing SAR data.

Polarimetric incoherent target decomposition aims at accessing physical parameters of illuminated scatters through the analysis of the target coherence or covariance matrix. In this framework, independent component analysis (ICA) was recently proposed as an alternative method to eigenvector decomposition to better interpret non-Gaussian heterogeneous clutter (inherent to high-resolution synthetic aperture radar systems). Until now, the two main drawbacks reported of the aforementioned method are the greater number of samples required for an unbiased estimation, when compared to the classical eigenvector decomposition, and the inability to be employed in scenarios under the Gaussian clutter assumption. In this paper, both drawbacks are analyzed. First, a Monte Carlo approach is performed in order to investigate the bias in estimating Touzi's target-scattering-vector-model parameters when ICA is employed. Simulated data and a RAMSES X-band image acquired over Brétigny, France, are taken into consideration to investigate the bias estimation under different scenarios. Finally, the performance of the algorithm is also evaluated under the Gaussian clutter assumption and when spatial correlation is introduced in the model.

New hyperspectral missions will collect huge amounts of hyperspectral data. In addition, it is possible now to acquire time series and multiangular hyperspectral images. The process and analysis of these big data collections will require common hyperspectral techniques to be adapted or reformulated. The tensor decomposition, which is also known as multiway analysis, is a technique to decompose multiway arrays, i.e., hypermatrices with more than two dimensions (ways). Hyperspectral time series and multiangular acquisitions can be represented as a three-way tensor. Here, we apply canonical polyadic (CP) tensor decomposition techniques to the blind analysis ohyperspectral big data. In order to do so, we use a novel compression-based nonnegative CP decomposition. We show that the proposed methodology can be interpreted as multilinear blind spectral unmixing, i.e., a higher order extension of the widely known spectral unmixing. In the proposed approach, the big hyperspectral tensor is decomposed in three sets of factors, which can be interpreted as spectral signatures, their spatial distribution, and temporal/angular changes. We provide experimental validation using a study case of the snow coverage of the French Alps during the snow season.

With different principles, support vector machines (SVMs) and multiple classifier systems (MCSs) have shown excellent performances for classifying hyperspectral remote sensing images. In order to further improve the performance, we propose a novel ensemble approach, namely, rotation-based SVM (RoSVM), which combines SVMs and MCSs together. The basic idea of RoSVM is to generate diverse SVM classification results using random feature selection and data transformation, which can enhance both individual accuracy and diversity within the ensemble simultaneously. Two simple data transformation methods, i.e., principal component analysis and random projection, are introduced into RoSVM. An empirical study on three hyperspectral data sets demonstrates that the proposed RoSVM ensemble method outperforms the single SVM and random subspace SVM. The impacts of the parameters on the overall accuracy of RoSVM (different training sets, ensemble sizes, and numbers of features in the subset) are also investigated in this paper.

In this paper, we consider the problem of unmixing a time series of hyperspectral images. We propose a dynamical model based on linear mixing processes at each time instant. The spectral signatures and fractional abundances of the pure materials in the scene are seen as latent variables, and assumed to follow a general dynamical structure. Based on a simplified version of this model, we derive an efficient spectral unmixing algorithm to estimate the latent variables by performing alternating minimizations. The performance of the proposed approach is demonstrated on synthetic and real multitemporal hyperspectral images.

In the above-named work, the support information in the first footnote is corrected.

Nonlinear decomposition schemes constitute an alternative to classical approaches for facing the problem of data fusion. In this paper, we discuss the application of this methodology to a popular remote sensing application called pansharpening, which consists in the fusion of a low resolution multispectral image and a high-resolution panchromatic image. We design a complete pansharpening scheme based on the use of morphological half gradient operators and demonstrate the suitability of this algorithm through the comparison with the state-of-the-art approaches. Four data sets acquired by the Pleiades, Worldview-2, Ikonos, and Geoeye-1 satellites are employed for the performance assessment, testifying the effectiveness of the proposed approach in producing top-class images with a setting independent of the specific sensor.

In image deconvolution problems, the diagonalization of the underlying operators by means of the fast Fourier transform (FFT) usually yields very large speedups. When there are incomplete observations (e.g., in the case of unknown boundaries), standard deconvolution techniques normally involve non-diagonalizable operators, resulting in rather slow methods or, otherwise, use inexact convolution models, resulting in the occurrence of artifacts in the enhanced images. In this paper, we propose a new deconvolution framework for images with incomplete observations that allows us to work with diagonalized convolution operators, and therefore is very fast. We iteratively alternate the estimation of the unknown pixels and of the deconvolved image, using, e.g., an FFT-based deconvolution method. This framework is an efficient, high-quality alternative to existing methods of dealing with the image boundaries, such as edge tapering. It can be used with any fast deconvolution method. We give an example in which a state-of-the-art method that assumes periodic boundary conditions is extended, using this framework, to unknown boundary conditions. Furthermore, we propose a specific implementation of this framework, based on the alternating direction method of multipliers (ADMM). We provide a proof of convergence for the resulting algorithm, which can be seen as a "partial" ADMM, in which not all variables are dualized. We report experimental comparisons with other primal-dual methods, where the proposed one performed at the level of the state of the art. Four different kinds of applications were tested in the experiments: deconvolution, deconvolution with inpainting, superresolution, and demosaicing, all with unknown boundaries.

Remote sensing hyperspectral images (HSIs) are quite often low rank, in the sense that the data belong to a low dimensional subspace/manifold. This has been recently exploited for the fusion of low spatial resolution HSI with high spatial resolution multispectral images in order to obtain super-resolution HSI. Most approaches adopt an unmixing or a matrix factorization perspective. The derived methods have led to state-of-the-art results when the spectral information lies in a low-dimensional subspace/manifold. However, if the subspace/manifold dimensionality spanned by the complete data set is large, i.e., larger than the number of multispectral bands, the performance of these methods mainly decreases because the underlying sparse regression problem is severely ill-posed. In this paper, we propose a local approach to cope with this difficulty. Fundamentally, we exploit the fact that real world HSIs are locally low rank, that is, pixels acquired from a given spatial neighborhood span a very low-dimensional subspace/manifold, i.e., lower or equal than the number of multispectral bands. Thus, we propose to partition the image into patches and solve the data fusion problem independently for each patch. This way, in each patch the subspace/manifold dimensionality is low enough, such that the problem is not ill-posed anymore. We propose two alternative approaches to define the hyperspectral super-resolution through local dictionary learning using endmember induction algorithms. We also explore two alternatives to define the local regions, using sliding windows and binary partition trees. The effectiveness of the proposed approaches is illustrated with synthetic and semi real data.

In the past 20 years, a wide range of complex fluoroscopically guided procedures have shown considerable growth. Biologic effects of the exposure (radiation induced burn, cancer) lead to reduce the dose during the intervention, for the safety of patients and medical staff. However, when the dose is reduced, image quality decreases, with a high level of noise and a very low contrast. Efficient restoration and denoising algorithms should overcome this drawback. We propose a spatio-temporal filter operating in a multi-scales space. This filter relies on a first order, motion compensated, recursive temporal denoising. Temporal high frequency content is first detected and then matched over time to allow for a strong denoising in the temporal axis. We study this filter in the curvelet domain and in the dual-tree complex wavelet domain, and compare those results to state of the art methods. Quantitative and qualitative analysis on both synthetic and real fluoroscopic sequences demonstrate that the proposed filter allows a great dose reduction.

We propose a temporal motion-compensated filter operating on dual-tree complex wavelet coefficients to denoise low-dose X-ray sequences. This filter allows for a great noise reduction while preserving moving objects and structures. We take advantage of the properties of multi-scale spaces to perform a fast and robust motion tracking. The result of this step is then used in the temporal filter for denoising purpose. Quantitative and qualitative analysis on real fluoroscopic sequences show that our method outperforms state-of-the-art VBM3D method and allows for a dose reduction as high as 80%.

The problem with visualization of hyper-spectral images on tri-stimulus displays arises from the fact that they contain hundreds of spectral bands while generally used display devices support only three bands/channels namely blue, green and red. Therefore, for visualization a hyper-spectral (HS) image has to be reduced to three bands. The main challenge while performing this band reduction is to retain and display the maximum information available in a hyper-spectral image. Human visual system focuses attention on certain regions in images called “salient regions”. Therefore to provide a comprehensive representation of hyper-spectral data on tri-stimulus displays we propose to use a weighted fusion method of saliency maps and hyper-spectral bands. The efficacy of the proposed algorithm has been demonstrated by tests on both urban and countryside images of AVIRIS and ROSIS sensors.

Pansharpening algorithms enable to produce synthetic data with high spatial details and spectral diversity by combining a panchromatic image with multispectral or hyperspectral data. In classical approaches the details extracted from the panchromatic image are introduced into the original multichannel image through injection gains, which can be spatially variant on the image. In this paper we analyze several methods for partitioning an image into regions in which the pixels will share the same injection coefficients. Gram-Schmidt pansharpening methods are used as paradigmatic examples for assessing the performance of global and local gain estimation strategies, using hyperspectral data acquired by sensors mounted on one (Earth Observing-1) or multiple (PROBA and Quick-bird) satellite platforms.

The Independent Component Analysis (ICA) aims, based on higher order statistical moments, in recovering statistical independent sources and the mixing mechanism, without having any physical background of the latter. Recently proposed as an alternative to Eigenvector decomposition in the analysis of Polarimetric SAR (PolSAR) data, it proved itself to be a very promising tool to better interpret non-Gaussian heterogeneous clutter, being employed in both urban area analysis as well as in snow monitoring applications. In this paper we intend to extend the range of applications of ICA based ICTD by investigating the results and the algorithm performance under tropical forest scenarios. Data from the P-band airborne dataset acquired by the Office National d'Études et de Recherches Aérospatiales (ONERA) over the French Guiana in 2009 in the frame of the European Space Agency campaign TropiSAR is taken into consideration to analyse the potential of supplementary information introduced by the ICA approach.

Pansharpening consists of fusing a multispectral (MS) image together with a panchromatic (PAN) image with the aim of jointly preserving the spectral diversity of the former and the geometric richness of the latter. A crucial step in pansharpening algorithms is the detail extraction. This problem is usually addressed by the means of 2D Gaussian filters matched with the MS sensor's modulation transfer function (MTF). Nevertheless, several issues can affect this characterization (e.g. the MTF's gains at the Nyquist frequency could be not available or unreliable). Thus, in this paper we propose a technique based on blind image deblurring in order to estimate band-dependent spatial detail extraction filters by taking into consideration the possible variability of the MS spatial features along bands. The validation is carried out exploiting two real datasets acquired by the IKONOS and the QuickBird sensors.

This letter introduces an alternative strategy for wet snow detection using multitemporal synthetic aperture radar (SAR) data. The proposed change detection method is primarily based on the comparison between two X-band SAR images acquired during the accumulation (winter) and melting (spring) seasons, in the French Alps. The new decision criterion relies on the local intensity statistics of the SAR images by considering the backscattering ratio as a stochastic process: the probability that “the intensity ratio fits into the predetermined range of values” is larger than a defined confidence level. Both the conducted snow backscattering simulations and the state-of-the-art measurements indicate more complex relation between the backscattering properties of the two snow types, with respect to the conventional assumption of the augmented electromagnetic absorption associated to the wet snow. Therefore, rather than adopting the standard hypothesis, we analyze the wet/dry snow backscattering ratio as a function of the local incidence angle (LIA). After employing the multilayer snow backscattering simulator, calibrated with scatterometer measurements in C-band, we modify, to some extent, the range of ratio values indicating the presence of the wet snow, by including positive ratio values for lower LIA. By simultaneously accounting for the speckle noise, the proposed stochastic approach derives the refined wet snow probability map. The performance analyses are carried out both through the comparison with the ground air temperature map and by comparing two copolarized channels processed separately.

In this letter, we propose a method aiming at reducing the noise in hyperspectral images based on the nonlinear generalization of principal component analysis (NLPCA). NLPCA is performed by an autoassociative neural network (AANN) that has the hyperspectral image as input and is trained to reconstruct the same image at the output. Due to its topology, characterized by a bottleneck layer, the nonlinear AANN forces the hyperspectral image to be projected in a lower dimensionality feature space by removing noise and both linear and nonlinear correlations between spectral bands. This process permits to obtain enhancements in terms of the quality of the reconstructed hyperspectral image. The results conducted on different hyperspectral images are qualitatively and quantitatively discussed and demonstrate the potentialities of the proposed method, as compared with similar approaches such as PCA and kernel PCA.

This letter presents a novel technique for the forecast of the ground horizontal irradiance (GHI) from satellite-based images. To enhance the forecast accuracy, spatial information in addition to temporal information has been considered. This produced an increase in the computational load of the forecast process. Dimensionality reduction techniques based on nonlinear principal component analysis (PCA) are used to project the original data set into low-dimension feature space. A multilayer feedforward neural network classifier is used to model the signal through a training operation involving past history of the considered spatiotemporal signal. Experiments have been carried out on two different data sets. Comparisons with classical forecasting techniques demonstrate that the introduction of the spatial information permits to obtain better short-term forecast measurements for all types of sky conditions. Moreover, further analysis demonstrates that, compared with linear PCA, the nonlinear PCA is more appropriate for dimensionality reduction of spatiotemporal GHI data set.

The application of sparse representation (SR) theory to the fusion of multispectral (MS) and panchromatic images is giving a large impulse to this topic, which is recast as a signal reconstruction problem from a reduced number of measurements. This letter presents an effective implementation of this technique, in which the application of SR is limited to the estimation of missing details that are injected in the available MS image to enhance its spatial features. We propose an algorithm exploiting the details self-similarity through the scales and compare it with classical and recent pansharpening methods, both at reduced and full resolution. Two different data sets, acquired by the WorldView-2 and IKONOS sensors, are employed for validation, achieving remarkable results in terms of spectral and spatial quality of the fused product.

In this letter, we propose a novel approach for improving Random Forest (RF) in hyperspectral image classification. The proposed approach combines the ensemble of features and the semisupervised feature extraction (SSFE) technique. The main contribution of our approach is to construct an ensemble of RF classifiers. In this way, the feature space is divided into several disjoint feature subspaces. Then, the feature subspaces induced by the SSFE technique are used as the input space to an RF classifier. This method is compared with a regular RF and an RF with the reduced features by the SSFE on two real hyperspectral data sets, showing an improved performance in ill-posed, poor-posed, and well-posed conditions. An additional study shows that the proposed method is less sensitive to the parameters.

Remote sensing is one of the most common ways to extract relevant information about Earth and our environment. Remote sensing acquisitions can be done by both active (synthetic aperture radar, LiDAR) and passive (optical and thermal range, multispectral and hyperspectral) devices. According to the sensor, a variety of information about the Earth's surface can be obtained. The data acquired by these sensors can provide information about the structure (optical, synthetic aperture radar), elevation (LiDAR), and material content (multispectral and hyperspectral) of the objects in the image. Once considered together their complementarity can be helpful for characterizing land use (urban analysis, precision agriculture), damage detection (e.g., in natural disasters such as floods, hurricanes, earthquakes, oil spills in seas), and give insights to potential exploitation of resources (oil fields, minerals). In addition, repeated acquisitions of a scene at different times allows one to monitor natural resources and environmental variables (vegetation phenology, snow cover), anthropological effects (urban sprawl, deforestation), climate changes (desertification, coastal erosion), among others. In this paper, we sketch the current opportunities and challenges related to the exploitation of multimodal data for Earth observation. This is done by leveraging the outcomes of the data fusion contests, organized by the IEEE Geoscience and Remote Sensing Society since 2006. We will report on the outcomes of these contests, presenting the multimodal sets of data made available to the community each year, the targeted applications, and an analysis of the submitted methods and results: How was multimodality considered and integrated in the processing chain? What were the improvements/new opportunities offered by the fusion? What were the objectives to be addressed and the reported solutions? And from this, what will be the next challenges?

This paper presents an alternative approach for polarimetric incoherent target decomposition (ICTD) dedicated to the analysis of very high-resolution polarimetric synthetic aperture radar (POLSAR) images. Given the non-Gaussian nature of the heterogeneous POLSAR clutter due to the increase in spatial resolution, the conventional methods based on the eigenvector target decomposition can ensure uncorrelation of the derived backscattering components at most. By introducing the independent component analysis (ICA) in lieu of the eigenvector decomposition, our method is rather deriving statistically independent components. The adopted algorithm, i.e., FastICA, uses the non-Gaussianity of the components as the criterion for their independence. Considering the eigenvector decomposition as being analogs to the principal component analysis (PCA), we propose the generalization of the ICTD methods to the level of the blind source separation (BSS) techniques (comprising both PCA and ICA). The proposed method preserves the invariance properties of the conventional ones, appearing to be robust both with respect to the rotation around the line of sight and to the change of the polarization basis. The efficiency of the method is demonstrated comparatively using POLSAR RAMSES X-band and ALOS L-band data sets. The main differences with respect to the conventional methods are mostly found in the behavior of the second most dominant component, which is not necessarily orthogonal to the first one. The potential of retrieving nonorthogonal mechanisms is moreover demonstrated using synthetic data. On the expense of a negligible entropy increase, the proposed method is capable of retrieving the edge diffraction of an elementary trihedral by recognizing dipole as the second component.

This paper presents a new algorithm for the analysis of linear spectral mixtures in the thermal infrared domain, with the goal to jointly estimate the abundance and the subpixel temperature in a mixed pixel, i.e., to estimate the relative proportion and the temperature of each material composing the mixed pixel. This novel approach is a two-step procedure. First, it estimates the emissivity and the temperature over pure pixels using the standard temperature and emissivity separation (TES) algorithm. Second, it estimates the abundance and the subpixel temperature using a new unmixing physics-based model, called Thermal Remote sensing Unmixing for Subpixel Temperature (TRUST). This model is based on an estimator of the subpixel temperature obtained by linearizing the black body law around the mean temperature of each material. The abundance is then retrieved by minimizing the reconstruction error with the estimation of the subpixel temperatures. The TRUST method is benchmarked on simulated scenes against the fully constrained least squares unmixing applied on the radiance and on the estimation of surface emissivity using the TES algorithm. The TRUST method shows better results on pure and mixed pixels composed of two materials. TRUST also shows promising results when applied on thermal hyperspectral data acquired with the Thermal Airborne Spectrographic Imager during the Detection in Urban scenario using Combined Airborne imaging Sensors campaign and estimates coherent localization of mixed-pixel areas.

Spectral unmixing is a popular technique for analyzing remotely sensed hyperspectral data sets with subpixel precision. Over the last few years, many algorithms have been developed for each of the main processing steps involved in spectral unmixing (SU) under the LMM assumption: 1) estimation of the number of endmembers; 2) identification of the spectral signatures of the endmembers; and 3) estimation of the abundance of endmembers in the scene. Although this general processing chain has proven to be effective for unmixing certain types of hyperspectral images, it also has some drawbacks. The first one comes from the fact that the output of each stage is the input of the following one, which favors the propagation of errors within the unmixing chain. A second problem is the huge variability of the results obtained when estimating the number of endmembers of a hyperspectral scene with different state-of-the-art algorithms, which influences the rest of the process. A third issue is the computational complexity of the whole process. To address the aforementioned issues, this paper develops a novel negative abundance-oriented SU algorithm that covers, for the first time in the literature, the main steps involved in traditional hyperspectral unmixing chains. The proposed algorithm can also be easily adapted to a scenario in which the number of endmembers is known in advance and two additional variations of the algorithm are provided to deal with high-noise scenarios and to significantly reduce its execution time, respectively. Our experimental results, conducted using both synthetic and real hyperspectral scenes, indicate that the presented method is highly competitive (in terms of both unmixing accuracy and computational performance) with regard to other SU techniques with similar requirements, while providing a fully self-contained unmixing chain without the need for any input parameters.

Hyperspectral remote sensing images (HSIs) usually have high spectral resolution and low spatial resolution. Conversely, multispectral images (MSIs) usually have low spectral and high spatial resolutions. The problem of inferring images that combine the high spectral and high spatial resolutions of HSIs and MSIs, respectively, is a data fusion problem that has been the focus of recent active research due to the increasing availability of HSIs and MSIs retrieved from the same geographical area. We formulate this problem as the minimization of a convex objective function containing two quadratic data-fitting terms and an edge-preserving regularizer. The data-fitting terms account for blur, different resolutions, and additive noise. The regularizer, a form of vector total variation, promotes piecewise-smooth solutions with discontinuities aligned across the hyperspectral bands. The downsampling operator accounting for the different spatial resolutions, the nonquadratic and nonsmooth nature of the regularizer, and the very large size of the HSI to be estimated lead to a hard optimization problem. We deal with these difficulties by exploiting the fact that HSIs generally “live” in a low-dimensional subspace and by tailoring the split augmented Lagrangian shrinkage algorithm (SALSA), which is an instance of the alternating direction method of multipliers (ADMM), to this optimization problem, by means of a convenient variable splitting. The spatial blur and the spectral linear operators linked, respectively, with the HSI and MSI acquisition processes are also estimated, and we obtain an effective algorithm that outperforms the state of the art, as illustrated in a series of experiments with simulated and real-life data.

Pansharpening aims at fusing a multispectral and a panchromatic image, featuring the result of the processing with the spectral resolution of the former and the spatial resolution of the latter. In the last decades, many algorithms addressing this task have been presented in the literature. However, the lack of universally recognized evaluation criteria, available image data sets for benchmarking, and standardized implementations of the algorithms makes a thorough evaluation and comparison of the different pansharpening techniques difficult to achieve. In this paper, the authors attempt to fill this gap by providing a critical description and extensive comparisons of some of the main state-of-the-art pansharpening methods. In greater details, several pansharpening algorithms belonging to the component substitution or multiresolution analysis families are considered. Such techniques are evaluated through the two main protocols for the assessment of pansharpening results, i.e., based on the full- and reduced-resolution validations. Five data sets acquired by different satellites allow for a detailed comparison of the algorithms, characterization of their performances with respect to the different instruments, and consistency of the two validation procedures. In addition, the implementation of all the pansharpening techniques considered in this paper and the framework used for running the simulations, comprising the two validation procedures and the main assessment indexes, are collected in a MATLAB toolbox that is made available to the community.

Many powerful pansharpening approaches exploit the functional relation between the fusion of PANchromatic (PAN) and MultiSpectral (MS) images. To this purpose, the modulation transfer function of the MS sensor is typically used, being easily approximated as a Gaussian filter whose analytic expression is fully specified by the sensor gain at the Nyquist frequency. However, this characterization is often inadequate in practice. In this paper, we develop an algorithm for estimating the relation between PAN and MS images directly from the available data through an efficient optimization procedure. The effectiveness of the approach is validated both on a reduced scale data set generated by degrading images acquired by the IKONOS sensor and on full-scale data consisting of images collected by the QuickBird sensor. In the first case, the proposed method achieves performances very similar to that of the algorithm that relies upon the full knowledge of the degrading filter. In the second, it is shown to outperform several very credited state-of-the-art approaches for the extraction of the details used in the current literature.

In this paper, we propose a new spectral-spatial classification strategy to enhance the classification performances obtained on hyperspectral images by integrating rotation forests and Markov random fields (MRFs). First, rotation forests are performed to obtain the class probabilities based on spectral information. Rotation forests create diverse base learners using feature extraction and subset features. The feature set is randomly divided into several disjoint subsets; then, feature extraction is performed separately on each subset, and a new set of linear extracted features is obtained. The base learner is trained with this set. An ensemble of classifiers is constructed by repeating these steps several times. The weak classifier of hyperspectral data, classification and regression tree (CART), is selected as the base classifier because it is unstable, fast, and sensitive to rotations of the axes. In this case, small changes in the training data of CART lead to a large change in the results, generating high diversity within the ensemble. Four feature extraction methods, including principal component analysis (PCA), neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA), and linearity preserving projection (LPP), are used in rotation forests. Second, spatial contextual information, which is modeled by MRF prior, is used to refine the classification results obtained from the rotation forests by solving a maximum a posteriori problem using the α-expansion graph cuts optimization method. Experimental results, conducted on three hyperspectral data with different resolutions and different contexts, reveal that rotation forest ensembles are competitive with other strong supervised classification methods, such as support vector machines. Rotation forests with local feature extraction methods, including NPE, LLTSA, and LPP, can lead to higher classification accuracies than that achieved by PCA. With the help of MRF, the proposed algorithms can improve the classification accuracies significantly, confirming the importance of spatial contextual information in hyperspectral spectral-spatial classification.

Classification is one of the most important techniques to the analysis of hyperspectral remote sensing images. Nonetheless, there are many challenging problems arising in this task. Two common issues are the curse of dimensionality and the spatial information modeling. In this paper, we present a new general framework to train series of effective classifiers with spatial information for classifying hyperspectral data. The proposed framework is based on the two key observations: 1) the curse of dimensionality and the high feature-to-instance ratio can be alleviated by using random subspace (RS) ensembles; and 2) the spatial-contextual information is modeled by the extended multiattribute profiles (EMAPs). Two fast learning algorithms, i.e., decision tree (DT) and extreme learning machine (ELM), are selected as the base classifiers. Six RS ensemble methods, namely, RS with DT, random forest (RF), rotation forest, rotation RF (RoRF), RS with ELM (RSELM), and rotation subspace with ELM (RoELM), are constructed by the multiple base learners. Experimental results on both simulated and real hyperspectral data verify the effectiveness of the RS ensemble methods for the classification of both spectral and spatial information (EMAPs). On the University of Pavia Reflective Optics Spectrographic Imaging System image, our proposed approaches, i.e., both RSELM and RoELM with EMAPs, achieve the state-of-the-art performances, which demonstrates the advantage of the proposed methods. The key parameters in RS ensembles and the computational complexity are also investigated in this paper.

Image guided interventions have seen growing interest in recent years. The use of X-rays for the procedure impels limiting the dose over time. Image sequences obtained thereby exhibit high levels of noise and very low contrasts. Hence, the development of efficient methods to enable optimal visualization of these sequences is crucial. We propose an original denoising method based on the curvelet transform. First, we apply a recursive temporal filter to the curvelet coefficients. As some residual noise remains, a spatial filtering is performed in the second step, which uses a magnitude-based classification and a contextual comparison of curvelet coefficients. This procedure allows to denoise the sequence while preserving low-contrasted structures, but does not improve their contrast. Finally, a third step is carried out to enhance the features of interest. For this, we propose a line enhancement technique in the curvelet domain. Indeed, thin structures are sparsely represented in that domain, allowing a fast and efficient detection. Quantitative and qualitative evaluations performed on synthetic and real low-dose sequences demonstrate that the proposed method enables a 50% dose reduction.

The estimation of surface emissivity and temperature from thermal hyperspectral data is a challenge. Methods that estimate the temperature and emissivity on a pixel composed by one single material exist. However, the estimation of the temperature on a mixed pixel, i.e. a pixel composed by more than one material, is more complex and has scarcely been investigated in the literature. This paper addresses this issue by proposing an estimator which linearizes the Black Body law around the mean temperature of each material. The performance of this estimator is studied using simulated data with different hyperspectral sensor configurations and under various noise conditions. The obtained results are encouraging and show an accuracy on the estimated temperature of 0.5 K while using high spectral resolution sensor.

Although the empirical mode decomposition is a powerful tool for analyzing complicated datasets, many irrelevant intrinsic mode functions may appear in the decomposition. In this paper, we develop an energy-based method to detect relevant intrinsic mode functions. The new method can be seen as a generalization of techniques that are based on correlation. An experimental study is carried out in different datasets for assessing the performance of the proposed technique.

In this manuscript, we propose a novel nonparametric test for nonstationarities that are seen as a trend or an evolution in the local energy of the signal. The idea of the proposed technique consists in applying empirical mode decomposition for estimating and further quantifying the trend in the time marginal of the estimated time-frequency representation. Such methodology allows for the detection of slowly-varying nonstationarities of first and second-order.

Path operators are powerful tools for the enhancement of thin and elongated objects in an image. In order to cope with noisy acquisition a variant of the path operators was recently proposed. However, both approaches cannot properly handle thin objects with tortuous shapes since strong variations of an object curvature produce disconnections in the paths. In order to address this issue, we propose a novel operator able to properly handle paths in tortuous shapes. It relies on the coupling of attribute filters based on the geodesic tortuosity and conventional path operators. Analogously to the complete version of the path operators, by allowing disconnections within paths it is possible also to define a path operator that is both robust and selective. The effectiveness of the proposed operators in filtering thin and tortuous image objects is proved on a 2D and 3D biomedical image.

The estimation of surface emissivity and temperature from thermal hyperspectral data is a challenge. There are several methods that estimate the temperature and the emissivity by assuming that the pixel is composed by a single material. However, the estimation of the temperature on a mixed pixel, i.e. a pixel composed by more than one material, is more complex and has scarcely been investigated in the literature. This paper addresses this issue by jointly estimating the materials composing the mixed pixel and their temperatures. It uses an unmixing method based on the linearization of the Black Body law. The performance of this strategy is studied using synthetic data and a real thermal image acquired by the TASI sensor.

We propose an image denoising method which takes curvelet domain inter-scale, inter-location and inter-orientation dependencies into account in a maximum a posteriori labeling of the curvelet coefficients of a noisy image. The rationale is that generalized neighborhoods of curvelet coefficients contain more reliable information on the true image than individual coefficients. Based on the labeling of coefficients and their magnitudes, a smooth thresholding functional produces denoised coefficients from which the denoised image is reconstructed. We also outline a faster approach to labeling and thresholding, relying on contextual comparisons of coefficients. Quantitative and qualitative evaluations on natural and X-ray images show that our method outperforms related multiscale approaches and compares favorably to the state-of-art BM3D method on X-ray data while executing faster.

Transform-based lossy compression has a huge potential for hyperspectral (HS) data reduction. The emerging JPEG2000 technology is based on the synergistic use of both spectral and spatial compression techniques. In this context the choice of the spectral decorrelation approach can have a strong impact on the quality of the compressed image. Since hyperspectral images are highly correlated within each spectral band and in particular across neighboring frequency bands, the choice of a spectral decorrelation method that allows to retain as much information content as possible is desirable. From this point of view, several methods based on PCA and Wavelet have been presented in the literature. In this paper, we propose the use of Nonlinear Principal Component Analysis (NLPCA) transform as a lossy spectral compression method applied to hyperspectral data. Being the NLPCA the nonlinear generalization of the standard principal component analysis (PCA), it permits to represent in a lower dimensional space the same information content with less features than the standard PCA.

In this paper, we propose a new method aiming at reducing the noise in hyperspectral images. It is based on the nonlinear generalization of Principal Component Analysis (NLPCA). The NLPCA is performed by an autoassociative neural network that have the hyperspectral image as input and is trained to reconstruct the same image at the output. Thanks to its bottleneck structure, the AANN forces the hyperspectral image to be projected in a lower dimensionality feature space where noise as well as both linear and nonlinear correlations between spectral bands are removed. This process permits to obtain enhancements in terms of hyperspectral image quality. Experiments are conducted on different real hyperspectral images, with different contexts and resolutions. The results are qualitatively and quantitatively discussed and demonstrate the interest of the proposed method as compared to traditional approaches.

Pansharpening is a successful application of data fusion to remotely sensed data. It aims at obtaining a detailed representation of an Earth's zone both in terms of spatial and spectral resolution. This is done through the fusion of a panchromatic and a multispectral image (having complementary spatial and spectral resolutions) that are acquired simultaneously by several optical satellites. The result of the fusion is commonly achieved by introducing the spatial details, modulated opportunely by gains, in the multispectral one. The injection gains can be estimated globally over the image, or locally, thus obtaining spatially variant values. The latter approach has been proven to achieve better results and it is based on windowing the analyzed image in squared blocks. In this paper we propose a more elaborated concept of locality, as it is based on an opportune segmentation of the target scene. In greater details, we propose to estimate the local injection gains on regions composed of pixel with similar spectral characteristic, as defined by a segmentation. Such local approach is compared to the global one and to the conventional local estimation based on overlapping and non-overlapping blocks. The performances have been assessed by using three real datasets, the first acquired by WorldView-2 and the other two by Pléiades. The analysis evidences the appreciable improvements of the performances with respect to classical schemes.

Hyperspectral remote sensing images (HSIs) are characterized by having a low spatial resolution and a high spectral resolution, whereas multispectral images (MSIs) are characterized by low spectral and high spatial resolutions. These complementary characteristics have stimulated active research in the inference of images with high spatial and spectral resolutions from HSI-MSI pairs. In this paper, we formulate this data fusion problem as the minimization of a convex objective function containing two data-fitting terms and an edge-preserving regularizer. The data-fitting terms are quadratic and account for blur, different spatial resolutions, and additive noise; the regularizer, a form of vector Total Variation, promotes aligned discontinuities across the reconstructed hyperspectral bands. The optimization described above is rather hard, owing to its non-diagonalizable linear operators, to the non-quadratic and non-smooth nature of the regularizer, and to the very large size of the image to be inferred. We tackle these difficulties by tailoring the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) - an instance of the Alternating Direction Method of Multipliers (ADMM) - to this optimization problem. By using a convenient variable splitting and by exploiting the fact that HSIs generally “live” in a low-dimensional subspace, we obtain an effective algorithm that yields state-of-the-art results, as illustrated by experiments.

The Reed-Xiaoli (RX) is considered as the benchmark algorithm in multidimensional anomaly detection (AD). However, the RX detector performance decreases when the statistical parameters estimation is poor. This could happen when the background is non-homogeneous or the noise independence assumption is not fulfilled. For a better performance, the statistical parameters are estimated locally using a sliding window approach. In this approach, called adaptive RX, a window is centered over the pixel under the test (PUT), so the background mean and covariance statistics are estimated using the data samples lying inside the window's spatial support, named the secondary data. Sometimes, a smaller guard window prevents those pixels close to the PUT to be used, in order to avoid the presence of outliers in the statistical estimation. The size of the window is chosen large enough to ensure the invertibility of the covariance matrix and small enough to justify both spatial and spectral homogeneity. We present here an alternative methodology to select the secondary data for a PUT by means of a binary partition tree (BPT) representation of the image. We test the proposed BPT-based adaptive hyperspectral RX AD algorithm using a real dataset provided by the Target Detection Blind Test project.

Remote sensing hyperspectral images (HSI) are quite often locally low rank, in the sense that the spectral vectors acquired from a given spatial neighborhood belong to a low dimensional subspace/manifold. This has been recently exploited for the fusion of low spatial resolution HSI with high spatial resolution multispectral images (MSI) in order to obtain super-resolution HSI. Most approaches adopt an unmixing or a matrix factorization perspective. The derived methods have led to state-of-the-art results when the spectral information lies in a low dimensional subspace/manifold. However, if the subspace/manifold dimensionality spanned by the complete data set is large, the performance of these methods decrease mainly because the underlying sparse regression is severely ill-posed. In this paper, we propose a local approach to cope with this difficulty. Fundamentally, we exploit the fact that real world HSI are locally low rank, to partition the image into patches and solve the data fusion problem independently for each patch. This way, in each patch the subspace/manifold dimensionality is low enough to obtain useful super-resolution. We explore two alternatives to define the local regions, using sliding windows and binary partition trees. The effectiveness of the proposed approach is illustrated with synthetic and semi-real data.

This paper presents an elaboration of the ICA based ICTD, proposed in [1]. The method is applied on three different datasets and three distinctive aspects of its performances are considered. Firstly, we challenge the initial choice of the ICA algorithm, by testing the suitability of two representative tensorial (fourth-order) and one second-order algorithm. Further, we demonstrate the invariance of the proposed decomposition with respect to both the rotation around the line of sight and the change of polarisation basis. Finally, we analyse the potential of supplementary information contained in the second most dominant component.

Anomaly Detection methods are used when there is not enough information about the target to detect. These methods search for pixels in the image with spectral characteristics that differ from the background. The most widespread detection test, the RX-detector, is based on the Mahalanobis distance and on the background statistical characterization through the mean vector and the covariance matrix. Although non-Gaussian distributions have already been introduced for background modeling in Hyperspectral Imaging, the parameters estimation is still performed using the Maximum Likelihood Estimates for Gaussian distribution. This paper describes robust estimation procedures more suitable for non-Gaussian environment. Therefore, they can be used as plug-in estimators for the RX-detector leading to some great improvement in the detection process. This theoretical improvement has been evidenced over two real hyperspectral images.

The snow coverage area (SCA) is one of the most important parameters for cryospheric studies. The use of remote sensing imagery can complement field measurements by providing means to derive SCA with a high temporal frequency and covering large areas. Images acquired by the Moderate Resolution Imaging Spectroradiometer (MODIS) are perhaps the most widely used data to retrieve SCA maps. Some MODIS derived algorithms are available for subpixel SCA estimation, as MODSCAG and MODImLab. Both algorithms make use of spectral unmixing techniques using a fixed set of snow, rocks and other materials spectra (endmembers). We aim to improve the performance of a modified version of MODIm-Lab algorithm by exploring advanced spectral unmixing techniques. Furthermore, we make use of endmember induction algorithms to obtain the endmembers from the data itself instead of using a fixed spectral library. We validate the proposed approach on a case study in the mountainous region of the Alps.

The design of a pansharpening algorithm for enriching a MultiSpectral image with the spatial details of a Panchromatic image should preserve the characteristics of the original dataset. A widely employed quality check consists in verifying the consistency of the fused product, namely the similarity of the original image and a reduced resolution version of the sharpened product. We propose to improve this feature by applying an Iterative Back-Projection algorithm after the fusion procedure. The approach is validated on two datasets, acquired by the Ikonos and WorldView-2 sensors, showing remarkable improvements, especially in conjunction with Component Substitution pansharpening methods.

In this paper state-of-the-art and advanced methods for multispectral pansharpening are reviewed and evaluated on two very high resolution datasets acquired by IKONOS-2 (four bands) and WorldView-2 (eight bands). The experimental analysis allows us to highlight the performances of the two main pansharpening approaches (i.e. component substitution and multiresolution analysis).

Images with high spatial and spectral resolutions are desirable for remote sensing applications. Unfortunately, due to sensor physical constraints, this result cannot be obtained by a single sensor. To overcome these limitations, a great number of data fusion approaches have been developed in the last years. The fusion of panchromatic and multispectral images, also known as Pansharpening, is capturing a lot of attention in the literature. In this paper, we extend and analyze the use of some classical pansharpening techniques, belonging to the MultiResolution Analysis and Component Substitution families, for fusing hyperspectral data instead of multispectral ones. The experimental results, conducted on two real datasets acquired by the Hyperion/ALI and CHRIS-Proba/QuickBird sensors, point out the greater suitability of the algorithms into the MRA class thanks to a better spectral consistency of the final products, which is a desirable feature when the number of bands to fuse increases.

The pansharpening process has the purpose of building a high-resolution multispectral image by fusing low spatial resolution multispectral and high-resolution panchromatic observations. A very credited method to pursue this goal relies upon the injection of details extracted from the panchromatic image into an upsampled version of the low-resolution multispectral image. In this letter, we compare two different injection methodologies and motivate the superiority of contrast-based methods both by physical consideration and by numerical tests carried out on remotely sensed data acquired by IKONOS and Quickbird sensors.

In this letter, an ensemble learning approach, Rotation Forest, has been applied to hyperspectral remote sensing image classification for the first time. The framework of Rotation Forest is to project the original data into a new feature space using transformation methods for each base classifier (decision tree), then the base classifier can train in different new spaces for the purpose of encouraging both individual accuracy and diversity within the ensemble simultaneously. Principal component analysis (PCA), maximum noise fraction, independent component analysis, and local Fisher discriminant analysis are introduced as feature transformation algorithms in the original Rotation Forest. The performance of Rotation Forest was evaluated based on several criteria: different data sets, sensitivity to the number of training samples, ensemble size and the number of features in a subset. Experimental results revealed that Rotation Forest, especially with PCA transformation, could produce more accurate results than bagging, AdaBoost, and Random Forest. They indicate that Rotation Forests are promising approaches for generating classifier ensemble of hyperspectral remote sensing.

In recent years, sparse representations have been widely studied in the context of remote sensing image analysis. In this paper, we propose to exploit sparse representations of morphological attribute profiles for remotely sensed image classification. Specifically, we use extended multiattribute profiles (EMAPs) to integrate the spatial and spectral information contained in the data. EMAPs provide a multilevel characterization of an image created by the sequential application of morphological attribute filters that can be used to model different kinds of structural information. Although the EMAPs' feature vectors may have high dimensionality, they lie in class-dependent low-dimensional subpaces or submanifolds. In this paper, we use the sparse representation classification framework to exploit this characteristic of the EMAPs. In short, by gathering representative samples of the low-dimensional class-dependent structures, any given sample may by sparsely represented, and thus classified, with respect to the gathered samples. Our experiments reveal that the proposed approach exploits the inherent low-dimensional structure of the EMAPs to provide state-of-the-art classification results for different multi/hyperspectral data sets.

Nonlinear spectral mixture models have recently received particular attention in hyperspectral image processing. In this paper, we present a novel optimization method of nonlinear unmixing based on a generalized bilinear model (GBM), which considers the second-order scattering of photons in a spectral mixture model. Semi-nonnegative matrix factorization (semi-NMF) is used for the optimization to process a whole image in matrix form. When endmember spectra are given, the optimization of abundance and interaction abundance fractions converge to a local optimum by alternating update rules with simple implementation. The proposed method is evaluated using synthetic datasets considering its robustness for the accuracy of endmember extraction and spectral complexity, and shows smaller errors in abundance fractions rather than conventional methods. GBM-based unmixing using semi-NMF is applied to the analysis of an airborne hyperspectral image taken over an agricultural field with many endmembers, and it visualizes the impact of a nonlinear interaction on abundance maps at reasonable computational cost.

The development of multisensor systems in recent years has led to great increase in the amount of available remote sensing data. Image fusion techniques aim at inferring high quality images of a given area from degraded versions of the same area obtained by multiple sensors. This paper focuses on pansharpening, which is the inference of a high spatial resolution multispectral image from two degraded versions with complementary spectral and spatial resolution characteristics: a) a low spatial resolution multispectral image; and b) a high spatial resolution panchromatic image. We introduce a new variational model based on spatial and spectral sparsity priors for the fusion. In the spectral domain we encourage low-rank structure, whereas in the spatial domain we promote sparsity on the local differences. Given the fact that both panchromatic and multispectral images are integrations of the underlying continuous spectra using different channel responses, we propose to exploit appropriate regularizations based on both spatial and spectral links between panchromatic and the fused multispectral images. A weighted version of the vector Total Variation (TV) norm of the data matrix is employed to align the spatial information of the fused image with that of the panchromatic image. With regard to spectral information, two different types of regularization are proposed to promote a soft constraint on the linear dependence between the panchromatic and the fused multispectral images. The first one estimates directly the linear coefficients from the observed panchromatic and low resolution multispectral images by Linear Regression (LR) while the second one employs the Principal Component Pursuit (PCP) to obtain a robust recovery of the underlying low-rank structure. We also show that the two regularizers are strongly related. The basic idea of both regularizers is that the fused image should have low-rank and preserve edge locations. We use a variation of the recently proposed Split Augmented Lagrangian Shrinkage (SALSA) algorithm to effectively solve the proposed variational formulations. Experimental results on simulated and real remote sensing images show the effectiveness of the proposed pansharpening method compared to the state-of-the-art.

The binary partition tree (BPT) is a hierarchical region-based representation of an image in a tree structure. The BPT allows users to explore the image at different segmentation scales. Often, the tree is pruned to get a more compact representation and so the remaining nodes conform an optimal partition for some given task. Here, we propose a novel BPT construction approach and pruning strategy for hyperspectral images based on spectral unmixing concepts. Linear spectral unmixing consists of finding the spectral signatures of the materials present in the image (endmembers) and their fractional abundances within each pixel. The proposed methodology exploits the local unmixing of the regions to find the partition achieving a global minimum reconstruction error. Results are presented on real hyperspectral data sets with different contexts and resolutions.

This work presents a novel spatio-temporal cellular automata-based filtering (STCAF) for image sequence denoising. Most of the methods using cellular automata (CA) for image denoising involve the manual design of the rules that define the behaviour of the automata. This is a complex and not straightforward operation. In order to tackle this problem, this paper proposes to use evolutionary methods to obtain the CA set of rules which produces the best possible denoising under different noise models or/and image sources. This is implemented using a spatio-temporal neighbourhood for each pixel, which significantly improves the results with respect to simple spatio or temporal set of neighbours. The proposed method is tested to reduce the noise in low-dose X-ray image sequences. These data have a severe signal-dependent noise that must be reduced avoiding artifacts while preserving structures of interest for a medical inspection. The proposed method outperforms several state-of-the-art algorithms on both simulated and real sequences.

The Binary Partition Tree (BPT) is a hierarchical region-based representation of an image in a tree structure. BPT allows users to explore the image at different segmentation scales, from fine partitions close to the leaves to coarser partitions close to the root. Often, the tree is pruned so the leaves of the resulting pruned tree conform an optimal partition given some optimality criterion. Here, we propose a novel BPT construction approach and pruning strategy for hyperspectral images based on spectral unmixing concepts. The proposed methodology exploits the local unmixing of the regions to find the partition achieving a global minimum reconstruction error. We successfully tested the proposed approach on the well-known Cuprite hyperspectral image collected by NASA Jet Propulsion Laboratory's Airborne Visible/Infrared Imaging Spectrometer (AVIRIS). This scene is considered as a standard benchmark to validate spectral unmixing algorithms.

The operational level reached by polarimetric data processing techniques has been demonstrated during the last decade. The next generation of spaceborne Synthetic Aperture Radar satellites will implement full- or dual- polarimetric capabilities. In few years a huge amount of data will have to be processed in a fast and reliable way, implementing polarimetric decompositions or accurate classifications. Two neural network approaches for fast and accurate processing of polarimetric data are presented. In the first approach a neural network based processing chain for fast model based polarimetric decomposition is developed, while in the second approach a Non-Linear Principal Component Analisys of polarimetric data has been performed using an Auto Associative Neural Network. The results show a considerable reduction of computational effort and a substantial data compression with a minimun loss of information.

This paper deals particularly with the sensitivity of the wet snow backscattering coefficient on density change. The presented backscattering model is based on the approach used in the dry snow analysis [1], appropriately modified to account for the increased dielectric contrast caused by liquid water presence. It encircles our undertaking of simulating and analysing snow backscattering using fundamental scattering theories (IEM-B, QCA, QCA-CP). The wet snow parameters are chosen according to the area of the particular interest - the French Alps, while the choice of the SAR sensor parameters (frequency, polarization) is primarily conditioned by the initially settled goal - reaching qualitative conclusions concerning wet snow backscattering mechanism. Based on simulation results, we state the dominance of the snow pack surface backscattering component, causing the backscattering to be directly proportional to the volumetric liquid water content. This result is confirmed by the performed in situ measurements. We illustrate as well the decrease of this effect with the increase in operating frequency.

This paper represents a part of our efforts to generalize polarimetric incoherent target decomposition to the level of BSS techniques by introducing the ICA method instead of the conventional eigenvector decomposition. We compare, in the frame of polarimetric incoherent target decomposition, several criteria for the estimation of complex independent components [1, 2]. This is done by parametrising the obtained dominant and mutually independent target vectors using the TSVM [3] and representing them on the corresponding Poincaré sphere. We demonstrate notably good performances of the proposed method applied on the RAMSES POLSAR X-band image, by precisely identifying the class of trihedral reflectors present in the scene. Logarithm and square root nonlinearities - two of the three proposed criteria for complex IC derivation prove to be very efficient. The best discrimination between the a priori defined classes appears to be achieved with the principal kurtosis criterion. Finally, the algorithm using the former two functions leads to very interesting entropy estimation.

Recently, a general approach for high-resolution polarimetric SAR (POLSAR) data classification in heterogeneous clutter was presented, based on a statistical test of equality of covariance matrices. Here, we extend that approach by taking advantage of the Constant False Alarm Ratio (CFAR) property of the statistical test in order to improve the clustering process. We show that the CFAR property can be used in the hierarchical segmentation of the POLSAR data images to automatically detect the number of clusters. The proposed method will be applied on a high-resolution polarimetric data set acquired by the ONERA RAMSES system.

When accounting for heterogeneity and non-Gaussianity of real hyperspectral data, elliptical distributions provide reliable models for background characterization. Through these assumptions, this paper highlights the fact that robust estimation procedures are an interesting alternative to classical methods and can bring some great improvement to the detection process. The goal of this paper is then not only to recall well-known methodologies of target detection but also to propose ways to extend them for taking into account the heterogeneity and non-Gaussianity of the hyperspectral images.

Real-time monitoring of agricultural crops is increasingly important because of the involved huge economic impact. The automatic identification of crops, as early as possible during the agricultural season, is an important issue supporting agricultural policies. In this context, the objective of this article is to evaluate the possibilities of remote sensing data to identify corn and soybean crops in the early growing season. The proposed study evaluates the potential of hyperspectral data and multispectral NDVI times series. Experimental results illustrate the challenges of crop detection in early stages.

In this work, an image representation based on Binary Partition Tree is proposed for object detection in hyperspectral images. The BPT representation defines a search space for constructing a robust object identification scheme. Spatial and spectral information are integrated in order to analyze hyperspectral images with a region-based perspective. Experimental results demonstrate the good performances of this BPT-based approach.

Polarimetry and multi-pass interferometry extend the dimensionality of SAR images, therefore the necessity to have multivariate statistic (and non-Gaussian) distributions as models for these types of data: such are the SIRV (Spherically Invariant Random Vectors). However, as the stochastic model becomes more complex, correctly estimating its parameters gets difficult. More, although they are versatile, the SIRV models are not guaranteed to match the PolSAR / InSAR data. To evaluate the pertinence of those models with respect to the PolSAR and multi-pass InSAR data, through one of their most important statistic properties, namely sphericity, it is the purpose of this paper. The proposed analysis is illustrated with spaceborne multi-pass InSAR TerraSAR-X data.

Remote sensors capture large scenes that are conventionally split in smaller patches before being stored and analyzed. Traditionally, this has been done by dividing the scene in rectangular windows. Such windowing methodology could provoke the separation of spectrally homogeneous areas or objects of interest into two or more patches. This is due to the presence of objects of interest in correspondence to windows' borders, or because the fixed size of the windows does not adapt well to the scale of the objects. To alleviate this issue, the windows can be arranged in an overlapping way, incurring in some data redundancy storage. Recently, tree representations have been used as an alternative to windowing in order to structure and store large amounts of remote sensing data. In this work we explore the benefits of using Binary Partition Trees (BPT) instead of windowing to store hyperspectral large scenes. We are particularly interested in storing the information resulting of local spectral unmixing processes running over a large real hyperspectral scene. We show that under similar conditions BPT allows a better storage of the unmixing information in terms of reconstruction error.

In this letter, an eigenvalue-based empirical method is proposed in order to estimate the number of endmembers in hyperspectral data. This method is based on the distribution of the differences of the eigenvalues from the correlation and the covariance matrices, respectively. The eigenvalues corresponding to the noise are identical in the covariance and the correlation matrices, while the eigenvalues corresponding to the signal (the endmembers) are larger in the correlation matrix than in the covariance matrix. The proposed method is totally parameter free and very fast. It is validated by experiments carried on both synthetic and real data sets.

Pansharpening may be defined as the process of synthesizing multispectral images at a higher spatial resolution. A wide range of pansharpening methods are available, each producing images with different characteristics. To compare the performances and characteristics of different methods, a contest was held in 2006 by the IEEE Data Fusion Technical Committee. In this contest, À trous wavelet transform-based pansharpening (AWLP) and Laplacian pyramid-based context adaptive (CBD) pansharpening methods were declared as joint winners. While assessing the quantitative quality of the pansharpened images, we observed that the two methods outperform each other depending upon the local content of the scene. Hence, it is interesting to design a method taking advantage of both methods by locally selecting the best one. This adaptive decision fusion is performed based on the local scale of the structure. The interest of the proposed method is verified using both visual and quantitative analyses for different Pléiades data sets.

The current increase of spatial as well as spectral resolutions of modern remote sensing sensors represents a real opportunity for many practical applications but also generates important challenges in terms of image processing. In particular, the spatial correlation between pixels and/or the spectral correlation between spectral bands of a given pixel cannot be ignored. The traditional pixel-based representation of images does not facilitate the handling of these correlations. In this paper, we discuss the interest of a particular hierarchical region-based representation of images based on binary partition tree (BPT). This representation approach is very flexible as it can be applied to any type of image. Here both optical and radar images will be discussed. Moreover, once the image representation is computed, it can be used for many different applications. Filtering, segmentation, and classification will be detailed in this paper. In all cases, the interest of the BPT representation over the classical pixel-based representation will be highlighted.

The articles in special issue focus on advancements in very high resolution remote sensing technologies and applications.

Recent advances in spectral-spatial classification of hyperspectral images are presented in this paper. Several techniques are investigated for combining both spatial and spectral information. Spatial information is extracted at the object (set of pixels) level rather than at the conventional pixel level. Mathematical morphology is first used to derive the morphological profile of the image, which includes characteristics about the size, orientation, and contrast of the spatial structures present in the image. Then, the morphological neighborhood is defined and used to derive additional features for classification. Classification is performed with support vector machines (SVMs) using the available spectral information and the extracted spatial information. Spatial postprocessing is next investigated to build more homogeneous and spatially consistent thematic maps. To that end, three presegmentation techniques are applied to define regions that are used to regularize the preliminary pixel-wise thematic map. Finally, a multiple-classifier (MC) system is defined to produce relevant markers that are exploited to segment the hyperspectral image with the minimum spanning forest algorithm. Experimental results conducted on three real hyperspectral images with different spatial and spectral resolutions and corresponding to various contexts are presented. They highlight the importance of spectral-spatial strategies for the accurate classification of hyperspectral images and validate the proposed methods.

With increased use of precision agriculture techniques, information concerning within-field crop yield variability is becoming increasingly important for effective crop management. Despite the commercial availability of yield monitors, many crop harvesters are not equipped with them. Moreover, yield monitor data can only be collected at harvest and used for after-season management. On the other hand, remote sensing imagery obtained during the growing season can be used to generate yield maps for both within-season and after-season management. This paper gives an overview on the use of airborne multispectral and hyperspectral imagery and high-resolution satellite imagery for assessing crop growth and yield variability. The methodologies for image acquisition and processing and for the integration and analysis of image and yield data are discussed. Five application examples are provided to illustrate how airborne multispectral and hyperspectral imagery and high-resolution satellite imagery have been used for mapping crop yield variability. Image processing techniques including vegetation indices, unsupervised classification, correlation and regression analysis, principal component analysis, and supervised and unsupervised linear spectral unmixing are used in these examples. Some of the advantages and limitations on the use of different types of remote sensing imagery and analysis techniques for yield mapping are also discussed.

Hyperspectral imagery, which contains hundreds of spectral bands, has the potential to better describe the biological and chemical attributes on the plants than multispectral imagery and has been evaluated in this paper for the purpose of crop yield estimation. The spectrum of each pixel in a hyperspectral image is considered as a linear combinations of the spectra of the vegetation and the bare soil. Recently developed linear unmixing approaches are evaluated in this paper, which automatically extracts the spectra of the vegetation and bare soil from the images. The vegetation abundances are then computed based on the extracted spectra. In order to reduce the influences of this uncertainty and obtain a robust estimation results, the vegetation abundances extracted on two different dates on the same fields are then combined. The experiments are carried on the multidate hyperspectral images taken from two grain sorghum fields. The results show that the correlation coefficients between the vegetation abundances obtained by unsupervised linear unmixing approaches are as good as the results obtained by supervised methods, where the spectra of the vegetation and bare soil are measured in the laboratory. In addition, the combination of vegetation abundances extracted on different dates can improve the correlations (from 0.6 to 0.7).

The optimal exploitation of the information provided by hyperspectral images requires the development of advanced image-processing tools. This paper proposes the construction and the processing of a new region-based hierarchical hyperspectral image representation relying on the binary partition tree (BPT). This hierarchical region-based representation can be interpreted as a set of hierarchical regions stored in a tree structure. Hence, the BPT succeeds in presenting: 1) the decomposition of the image in terms of coherent regions, and 2) the inclusion relations of the regions in the scene. Based on region-merging techniques, the BPT construction is investigated by studying the hyperspectral region models and the associated similarity metrics. Once the BPT is constructed, the fixed tree structure allows implementing efficient and advanced application-dependent techniques on it. The application-dependent processing of BPT is generally implemented through a specific pruning of the tree. In this paper, a pruning strategy is proposed and discussed in a classification context. Experimental results on various hyperspectral data sets demonstrate the interest and the good performances of the BPT representation.

Recently, a time-frequency approach for testing stationarity was proposed. However, this method inefficiently detects nonstationarities of the first-order. Here, we present two contributions that improve the test performance and allow the detection of first-order evolutions. The first one is to use an adequate distance measure. The second is a modification of the method in order to consider the spectral content from the signal itself when computing the distances.

Generally, for optical satellite sensors spatial and spectral resolutions are highly correlated factors. In fact, given the design constraints of these sensors, there is an inverse relation between their spatial and spectral resolution. Thus, the hyperspectral sensors have a high spectral resolution i.e. large number of bands covering the electromagnetic spectrum, but a lower spatial resolution. On the other hand, panchromatic (PAN) images have the highest spatial resolution but no spectral diversity. For better utilization and interpretation, hyperspectral images having both high spectral and spatial resolution are desired. This can be achieved by making use of a high spatial resolution PAN image in the context of pansharpening or image fusion. Several fusion approaches have been proposed in the literature. In this paper we propose the use of a hybrid algorithm combining substitution and injection methods. One of the main challenges in hyperspectral image fusion is the improvement of the spatial resolution, i.e. spatial details while preserving the original spectral information. This requires addition of pertinent spatial details to each band of the HS image. However, due to large number of bands the pansharpening of HS images is computationally expensive. Thus a dimensionality reduction preprocess, compressing the original number of measurements into a lower dimensional space, becomes mandatory. In this paper we propose the use of non-linear principal components instead of the original HS bands as input to a fusion process to enhance the spatial resolution of the HS image.

This paper provides comprehensive analysis of the dry snow pack backscattering coefficient dependence on the density change, for various SAR sensor parameters and chosen dry snow pack parameters, characteristic for the region of French Alps. As the result, qualitative conclusions, based on applying fundamental scattering theories (Rayleigh scattering model, Quasi Crystalline Approximation, Integral Equation Model) on the particular distributed target are presented. They represent the ground for semi-empirical models, which may provide a satisfactory link between backscattering coefficient and snow density (as one of the quantities defining SWE), as well as the guidelines for the further radar acquisitions over the Alpine region in France.

This paper proposes the new method for wet snow mapping using SAR data. It represents a modified version of the existing Nagler's mapping method, based on winter/summer image comparison, which is considered as the classic one. Instead of the existing unique threshold, a variable threshold matrix (function of the local incidence angle for each pixel) is proposed, based on dry and wet snow backscattering simulation results. The new membership decision method (with the respect to the dry/snow classes) is introduced. It considers the intensity ratio as a stochastical process: the probability that “the intensity ratio is smaller than the corresponding dry/wet snow determined threshold” is larger than the desired confidence level.

Support vector machine (SVM) and Random Forest (RF) have been developed to improve the accuracy of hyperspectral remote sensing (HRS) image classification significantly in recent years. Due to the different characteristics and obvious diversity between SVM and RF, we propose two integration approaches which combine SVM and Random Forest to classify the HRS image. The proposed method called DWDCS is examined by two hyperspectral images and it can acquire the higher overall accuracy and also improve the accuracy of each classes. Experimental results indicate that the proposed approaches have a great deal of advantages in classifying HRS image.

The detection of hedges in very high spatial resolution remote sensing image is discussed in the paper. A spatial-spectral detector is proposed. The spatial information is modeled per pixel as the local orientation of the structure to which the pixel belongs. The local orientation is computed from the morphological directional profile built with a series of linear openings in several directions. These features are used as inputs to a support vector data description, a detection algorithm. Experimental results on a real satellite image show that the local orientation helps in discriminating hedges from other woody elements, which is not possible using the spectral information only.

When dealing with impulsive background echoes, Gaussian model is no longer pertinent. We study in this paper the class of elliptically contoured (EC) distributions. They provide a multivariate location-scatter family of distributions that primarily serve as long tailed alternatives to the multivariate normal model. They are proven to represent a more accurate characterization of HSI data than models based on the multivariate Gaussian assumption. For data in ℝk, robust proposals for the sample covariance estimate are the M-estimators. We have also analyzed the performance of an adaptive non- Gaussian detector built with these improved estimators. Constant False Alarm Rate (CFAR) is pursued to allow the detector independence of nuisance parameters and false alarm regulation.

In this paper, a new technique is presented to detect the river networks in simulated SWOT images. The proposed algorithm is based on a noise reduction step followed by a directional morphological filter. In this work, the speckle noise reduction has been achieved by using a Curvelet-based filter preserving the structures of interest. After the filtering task, a river contrast enhancement has been presented by using the Path-Opening filter. This morphological filtering has retained the curvilinear structures on the image independently of their orientation. Hence, the river detection has been possible by a simple thresholding on the Path-Opening result. The obtained results are evaluated using a visual inspection and a quantitative evaluation. The potential of the proposed algorithm has been evaluated by studying the robustness of the parameters.

In remote sensing, pansharpening refers to the technique that combines the complementary spectral and spatial resolution characteristics of a multispectral image and a panchromatic image, with the objective to generate a high-resolution color image. This paper presents a new pansharpening method based on the minimization of a variant of total variation. We consider the fusion problem as the colorization of each pixel in the panchromatic image. A new term concerning the gradient of the panchromatic image is introduced in the functional of total variation so as to preserve edges. Experimental results on IKONOS satellite images demonstrate the effectiveness of the proposed method.

In this paper we propose an efficient method for population density estimation using textons and k nearest neighbor classifier (k-NN). Leung Malik (LM) filter bank is used for texture extraction (textons) from Google Earth Satellite Images and classification into high, medium, low population density and non-populated areas. We have tested the proposed method for 5 different images of cities of Pakistan at high resolution. Comparison of our results with those obtained using Grey Level Co-occurrence Matrix (GLCM) are also presented, indicating the effectiveness of the proposed method.

Remote sensing is a widely-used utility in supporting multilateral environmental treaties such as the Water Framework Directive (WFD). Regarding the WFD most remote sensing applications aim on the assessment of the biochemical status of surface water, while the general detection of water networks is disregarded. Therefore, a methodology for the automatic extraction of river networks from multispectral satellite data is presented.

In the literature, for sake of simplicity it is usually assumed that the model ruling spectral mixture in a hyperspectral pixels is basically linear. However, in many real life cases the different materials are usually in intimate association, like sand grains, resulting in a nonlinear mixture. Unfortunately, modeling a nonlinear approach is not trivial, and a general procedure is still up to be found. Aim of this paper is to evaluate the potentialities of Nonlinear Principal Component Analysis (NLPCA) as an approach to perform a nonlinear unmixing for the unsupervised extraction and quantification of the end-members. From this point of view scope of this paper is to demonstrate that the NLPCs derived from the proposed process can be considered as end-members. To perform an accurate evaluation, the proposed algorithm has been tested on two different hyperspectral datasets and compared with other approaches found in the literature.

In this paper we propose a new approach for the improvement of the spatial resolution of hyperspectral image classification maps combining both spectral unmixing and pansharpening approaches. The main idea is to use a spectral unmixing algorithm based on neural networks to retrieve the abundances of the endmembers present in the scene, and then use the spatial information retrieved from the pansharpened image to find the location of each endmember within the enhanced pixel according to the endmembers abundances. The proposed approach has been applied both to real and synthetic datasets.

In this paper, the ability of unsupervised linear unmixing has been evaluated for the hyperspectral image taken on the south pole of the planet Mars by OMEGA instrument aborad MEX. State-of-art methods of the three steps are tested: i) estimation of the number of endmembers; ii) extraction of endmembers; ii) estimation of the abundances. According to the results, it has been found that the combination ELM-VCA-NCLS has the best performance for obtaining the spectral signatures and the abundances of the three principle chemical species (CO2ice, water ice and dust) on the south pole of the Mars.

In this paper, seismic vulnerability assessment is addressed under the umbrella of remote sensing. A study for estimating and evaluating information for assessing seismic vulnerability based on a building basis is presented. The proposed methodology utilizes the capabilities of remote sensing and combines in-situ data tested in the area of Grenoble (France). A map is estimated in agreement with in-situ data, as support information system for seismic risk in the context of building vulnerability assessment. In the methodology proposed, building attributes such as roof identification, building height and characteristic scale are extracted from very high resolution panchromatic data, and an accurate digital elevation model. Support vector machine regression is used to estimate building vulnerability and in-situ data are available for evaluation.

Individual tree crown delineation in tropical forests is of great interest for ecological applications. In this paper we propose a method for hyperspectral image segmentation based on binary tree partitioning. The initial partition is obtained from a watershed transformation in order to make the method computationally more efficient. Then we use a non-parametric region model based on histograms to characterize the regions and the diffusion distance to define the region merging order. The pruning strategy is based on the discontinuity of size increment observed when iteratively merging the regions. The segmentation quality is assessed visually and appears to perform well on most cases, but tree delineation could be improved by including structural information derived from LiDAR data.

Nonlinear spectral mixing models have recently been receiving attention in hyperspectral image processing. This work presents a novel optimization method for nonlinear unmixing based on a generalized bilinear model (GBM), which considers second-order scattering effects. Semi-nonnegative matrix factorization is used for optimization to process a whole image in a matrix form. The proposed method is applied to an airborne hyperspectral image with many endmembers and shows good performance both in unmixing quality and computational cost with simple implementation. The effect of endmember extraction on nonlinear unmixing is investigated and the impact of the nonlinearity on abundance maps is demonstrated.

Morphological profiles (MPs) have been proposed in recent literature as aiding tools to achieve better results for classification of remotely sensed data. MPs are in general built using features containing most of the information content of the data, such as the components derived from principal component analysis (PCA). Recently, nonlinear PCA (NLPCA), performed by autoassociative neural network, has emerged as a good unsupervised technique to fit the information content of hyperspectral data into few components. The aim of this letter is to investigate the classification accuracies obtained using extended MPs built from the features of NPCA. A comparison of the two approaches has been validated on two different data sets having different spatial and spectral resolutions/coverages, over the same ground truth, and also using two different classification algorithms. The results show that NLPCA permits one to obtain better classification accuracies than using linear PCA.

Advanced information processing and architectures will be needed to bridge the gap between the potential offered by the new generations of sensors and the needs of the end-users to actually face tomorrow's challenges in many applications with a very high societal impact. As remote sensing researchers and engineers, this is our passion, our charge, and our responsibility.

In this paper, a new method for supervised hyperspectral data classification is proposed. In particular, the notion of stochastic minimum spanning forest (MSF) is introduced. For a given hyperspectral image, a pixelwise classification is first performed. From this classification map, M marker maps are generated by randomly selecting pixels and labeling them as markers for the construction of MSFs. The next step consists in building an MSF from each of the M marker maps. Finally, all the M realizations are aggregated with a maximum vote decision rule in order to build the final classification map. The proposed method is tested on three different data sets of hyperspectral airborne images with different resolutions and contexts. The influences of the number of markers and of the number of realizations M on the results are investigated in experiments. The performance of the proposed method is compared to several classification techniques (both pixelwise and spectral-spatial) using standard quantitative criteria and visual qualitative evaluation.

Burst-type signals constitute an important class of transient signals, being used especially in the investigation of various physical environments by electric or acoustic means. An important issue in the analysis of this type of signals is their detection in time. In this paper, we propose a detection method that is based on the histogram of the phase space distributed over time. The method consists in representing the analyzed signal in phase space and, then, quantifying the recurrences of the trajectory obtained in this space. In this way, we derive a time - recurrence radius representation for the signal, that allows identification of positions and durations of the transients. Afterwards, we propose a method to obtain a detection curve starting from this representation of the signal. We also present here some results concerning the performance of our method in the presence of noise on both synthetic and real signals.

The Hierarchical SEGmentation (HSEG) algorithm, which is a combination of hierarchical step-wise optimization and spectral clustering, has given good performances for hyperspectral image analysis. This technique produces at its output a hierarchical set of image segmentations. The automated selection of a single segmentation level is often necessary. We propose and investigate the use of automatically selected markers for this purpose. In this paper, a novel Marker-based HSEG (M-HSEG) method for spectral-spatial classification of hyperspectral images is proposed. First, a map of markers is constructed using classification results. Then, a novel constrained M-HSEG algorithm is applied. The experimental results show that the proposed approach yields accurate segmentation and classification maps, and thus is attractive for hyperspectral image analysis.

A new method for supervised hyperspectral data classification is proposed. In particular, the notion of Stochastic Minimum Spanning Forests (MSFs) is introduced. For a given hyper-spectral image, a pixelwise classification is first performed. From this classification map, M marker maps are generated by randomly selecting pixels and labeling them as markers for the construction of MSFs. The next step consists in building an MSF from each of the M marker maps. Finally, all the M realizations are aggregated with a maximum vote decision rule, resulting in a final classification map. The experimental results presented on an AVIRIS image of the vegetation area show that the proposed approach yields accurate classification maps, and thus is attractive for hyperspectral data analysis.

The work presented here proposes a new Binary Partition Tree pruning strategy aimed at the segmentation of hyperspectral images. The BPT is a region-based representation of images that involves a reduced number of elementary primitives and therefore allows to design a robust and efficient segmentation algorithm. Here, the regions contained in the BPT branches are studied by recursive spectral graph partitioning. The goal is to remove subtrees composed of nodes which are considered to be similar. To this end, affinity matrices on the tree branches are computed using a new distance-based measure depending on canonical correlations relating principal coordinates. Experimental results have demonstrated the good performances of BPT construction and pruning.

A kernel adapted to the spectral dimension of hyperspectral images is proposed in this paper. A distance based on a statistical cluster model is used to construct a radial kernel. This class specific kernel realizes a compromise between a conventional Gaussian kernel and a Gaussian kernel on the first principal components of the considered class. An automatic gradient optimization is used to select the optimal hyperparameters. Experimental results on a real hyperspectral image show the kernel is effective compared to the conventional Gaussian kernel. Furthermoren the proposed kernel is less sensitive to one hyperparameter compared to the Gaussian kernel applied on the first principal components of the data.

Low spatial resolution is a major limitation for remote sensing classification, especially in a urban environment. In this work, we will focus on the simulation of urban area environment at a low spatial resolution, comparable to the new hyperspectral sensors that will be launched in the next few years. The aim is to better understand the possibility offered by the new sensors, in a challenging scenario like the one represented by a highly mixed image. Particular attention is placed on the characteristics of the sensor EnMap, produced by DLR. The experiments conducted on a real data set confirm the challenges posed by low spatial resolution when analyzing a urban environment.