Blind hyperspectral unmixing (HU) has advanced significantly with the emergence of deep learning-based methods. However, the localized operations of convolutional neural networks (CNNs) and the high computational demands of Transformers present challenges for blind HU. This necessitates the development of image-level unmixing methods capable of capturing long-range spatial-spectral dependencies with low computational demands. This letter proposes a cascaded spatial-spectral Mamba model, termed UNMamba, which leverages the strengths of Mamba to efficiently model long-range spatial-spectral dependencies with linear computational complexity, achieving superior image-level unmixing performance with small parameters and operations. Specifically, UNMamba first captures long-range spatial dependencies, followed by the extraction of global spectral features, forming long-range spatial-spectral dependencies, which are subsequently mapped into abundance maps. Then, the input image is reconstructed using the linear mixing model (LMM), incorporating weighted averages of multiple trainable random sequences and an endmember loss to learn endmembers. UNMamba is the first unmixing approach that introduces the state-space models (SSMs). Extensive experimental results demonstrate that, without relying on any endmember initialization techniques [such as vertex component analysis (VCA)], the proposed UNMamba achieves significantly high unmixing accuracy, outperforming state-of-the-art methods. Codes are available athttps://github.com/Preston-Dong/UNMamba.

Hyperspectral images (HSIs) typically have finer spectral resolution but coarser spatial resolution than multispectral images (MSIs). To obtain HSIs with enhanced spatial resolution, considerable emphasis has been placed on achieving hyperspectral super-resolution (SR) by fusing HSIs with MSIs in the same scene. However, most existing HSI-MSI fusion methods either rely on prior knowledge of degradation models or require sufficient training data, hindering their practicality and interpretability. This letter proposes a deep multistream network (DMSN) for HSI SR. Specifically, we introduce the Spa-DNet and the Spe-UNet modules to encode spatial and spectral transformations across resolutions. Furthermore, we design the Int-Net to achieve spatial and spectral information interaction, enhancing the model’s performance. Finally, the proposed approach enables high spatial and spectral resolution HSIs. Using the newly designed three-stage training strategy, the network parameters can exhibit the clear physical significance of the degradation process, thereby helping to ensure faithful reconstruction of the desired HSIs. Experimental results with real datasets demonstrate that the proposed DMSN performs better than other methods. The codes will be available athttps://github.com/yuanchaosu/dmsn-GRSL.

Unsupervised hyperspectral image super-resolution (HSI-SR) has recently emerged as a popular and active research topic in remote sensing data fusion. However, most methods neglect the non-local features of the data in representation learning, which limits their fusion performances. To overcome the issue, we propose a Graph Attention-based Fusion Network (FusGAT) in this letter. This approach first extracts local features from the input data using multi-scale convolutions, and then the graph attention mechanism is employed to model relationships between nodes in the spectral stream for deriving non-local features of the image and transferring them to the spatial stream. FusGAT will iteratively update the node connections and refine node embedding, facilitating the extraction of non-local features and enabling effective information flow between the streams. We conducted several experiments on two datasets to prove the effectiveness of the proposed method. The source code will be available at: https://github.com/yuanchaosu/FusGAT-GRSL.

The fusion of hyperspectral images (HSIs) and multispectral images (MSIs) is crucial for overcoming the limitations of low spatial resolution in HSI. Currently, supervised learning methods tend to yield satisfactory integration results when applied to data distributions similar to those of the training set; however, they often exhibit insufficient generalization when confronted with real-world application scenarios. In contrast, unsupervised methods exhibit good generalization capabilities; however, they typically require careful tuning of hyperparameters to achieve satisfactory results, primarily due to the lack of sufficiently clear training objectives. To fully leverage the advantages of both supervised and unsupervised learning, this letter proposes an unsupervised pretraining framework (UPFW) guided fusion approach, which effectively enhances the performance of HSI-MSI fusion by introducing low-resolution supervised pretraining and full-resolution unsupervised adaptive strategy. Specifically, in the first stage, the model adapts to the learning spatial and spectral degradation parameter; in the second stage, we propose an adaptive fusion network (ADFNet) and conduct supervised learning on low-resolution scale to obtain a pretrained fusion network model with a clear objective-oriented; in the third stage, we utilize the pretrained model for full-resolution unsupervised fusion, thereby enhancing the model’s generalization capabilities and applicability. Experimental results show that compared to traditional methods and other deep learning approaches, the proposed method achieves significant advantages in spectral fidelity and spatial detail recovery across multiple public datasets.

Depending on a large-scale paired dataset of low-resolution hyperspectral image (LrHSI), high-resolution multispectral image (HrMSI), and corresponding high-resolution hyperspectral image (HrHSI), the supervised paradigm has achieved impressive performance in the hyperspectral image super-resolution (HISR). However, the intrinsic data-intensive manner hinders its further application in real scenarios. Fortunately, deep image prior (DIP) allows us to achieve unsupervised super-resolution (SR) by solely utilizing degraded observations. However, its potential to accurately model complicated hyperspectral priors is still not fully exploited due to the following two factors: 1) existing methods tend to reconstruct the unknown HrHSI directly from a randomly generated noise, leaving it hard to leverage the scene-relevant information for prior learning and 2) the vanilla architecture is handcrafted for the generator network, which shows limitations in feature representation and thus fails to characterize the complicated image properties. To unleash the potential of DIP for the HISR task, we propose an enhanced DIP network, called EDIP-Net, by addressing the aforementioned impediments. Specifically, EDIP-Net is built with a two-stage four-component scheme, with a zero-shot learning (ZSL) stage for input image establishment and a deep image generation (DIG) stage for prior learning. First, we exploit the cross-scale spectral relationship inside the observations and thus design a degradation learning network to generate paired training samples from the observations themselves. As such, two image-coarse estimations are derived in a ZSL manner by learning an interactive spectral learning network. By replacing random noise with two estimations, we design a double U-shape architecture for the generator network to capture their hyperspectral prior, each independently generating one HrHSI candidate. Under this premise, we further propose a degradation-aware decision fusion strategy to integrate the optimal results in a pixel-to-pixel manner. Extensive experiments demonstrate our superiority in achieving high-quality SR performance. The code will be available athttps://github.com/JiaxinLiCAS.

Convolutional Neural Networks (CNNs) and Transformers have made remarkable achievements in hyperspectral image classification (HSIC). Unfortunately, CNN-based methods struggle to capture the contextual dependencies between pixels in HSIs, while Transformer-based methods suffer from quadratic computational complexity. Recently, the Mamba model has shown great potential as it can describe long-range dependencies between HSI pixels with linear computational complexity. Yet, Mamba still faces significant challenges in terms of global modeling. Inspired by the Mamba model framework and Transformers, a novel Dual-Branch Mamba-Like Linear Attention (DBMLLA) network is proposed for HSIC, achieving efficient global dependency modeling. Specifically, the proposed DBMLLA combines an embedding module, a Spatial-Spectral Mamba-Like Linear Attention (SS-MLLA) module, and a fusion module. In the embedding module, an absolute position embedding module is introduced for better extraction of global features. In the SS-MLLA module, we design the Spatial Mamba-Like Linear Attention (SpaMLLA) block and the Spectral Mamba-Like Linear Attention (SpeMLLA) block to exploit the spatial and spectral information of the HSI. In addition, SS-MLLA is improved by utilising Depthwise Separable Convolution (DSC) to enhance the model’s ability to extract deeper local feature information. Through experiments conducted on four public hyperspectral datasets, it is demonstrated that the proposed model consistently outperforms state-of-the-art approaches.

Hyperspectral imaging can capture abundant spectral information and reveal the spectral absorption properties of surface materials. Nevertheless, the tradeoff in spatial resolution reduces its capacity to represent surface object textures and structures; hyperspectral super-resolution (SR) technology is a viable solution to this problem. Yet, mainstream supervised methods depend on low-scale training data and specific data distributions, restricting their generalization capability and practicality in real scenarios. Although unsupervised methods remove the reliance on training data, they still face suboptimal reconstruction quality due to the absence of reference images and inaccuracies in degradation process estimation. Furthermore, bridging the performance gap between simulated datasets and real-world applications remains challenging. To this end, we propose a semi-supervised network that effectively couples unsupervised learning and supervised pretraining in multistage architecture, MCS-Net for short. The network consists of three key components: degradation information estimation (DIE), supervised fusion pretraining (SFP) at low resolution, and unsupervised image generation (UIG) at full resolution. The MCS-Net first estimates deep degradation information from input image pairs using DIE. It then applies supervised learning in SFP to construct a pretrained fusion function and its parameters from the input low-resolution data pairs and their fused outputs. Finally, the pretrained parameters from the previous stage are used to initialize the fusion network of UIG, which is then fine-tuned under the guidance of degradation parameters estimated by DIE, enabling the network to process full-resolution images effectively. Ablation experiments validated the effectiveness of each component. Moreover, the proposed MCS-Net outperforms the existing state of the art (SOTA) methods across six evaluation metrics in the simulation experiments, and the experimental results on real satellite data further validate the outstanding image fusion performance of MCS-Net and its potential for practical applications.

Spatiotemporal fusion (STF) technology effectively addresses the difficulty in obtaining high spatial and temporal resolution images due to compromises in satellite design. Deep learning algorithms are extensively applied in this field, but their performance is significantly impacted by hardware factors, such as the vast differences in sensor resolutions. To address this issue, we have fully considered the scale differences among data from various sources and proposed a two-stage coarse-to-fine STF approach, named SIFnet. SIFnet comprises two stages; the first stage concentrates on learning the scale differences between the data, while the second stage integrates feature maps that represent different spatial resolutions to impose constraints on the network’s learning. This approach enables the preservation of detailed information by reusing feature maps at different scales during the learning process. As a result, the mapping relationship with the real image is closer. In this letter, three datasets containing different ground feature characteristics are used for STF experiments. The results demonstrate that SIFnet can effectively utilize the concept of scale transformation to improve feature extraction and enhance the accuracy of reconstructed images. Compared with various advanced STF algorithms, SIFnet achieves an average improvement of 2% in structural similarity, proving the effectiveness of this method.

By fusing a low-resolution hyperspectral image (LrMSI) with an auxiliary high-resolution multispectral image (HrMSI), hyperspectral image super-resolution (HISR) can generate a high-resolution hyperspectral image (HrHSI) economically. Despite the promising performance achieved by deep learning (DL), there are still two challenges remaining to be solved. First, most DL-based methods heavily rely on large-scale training triplets, which reduces them to limited generalization and poor practicability in real-world scenarios. Second, existing methods pursue higher performance by designing complex structures from off-the-shelf components while ignoring inherent information from the degradation model, hence leading to insufficient integration of domain knowledge and lower interpretability. To address those drawbacks, we propose a model-informed multi-stage unsupervised network, M2U-Net for short, by leveraging both deep image prior (DIP) and degradation model information. Generally, M2U-Net is built with a three-stage scheme, i.e., degradation information learning (DIL), initialized image establishment (IIE), and deep image generation (DIG) stages. The first stage is to exploit the deep information of the degradation model via a tiny network whose parameters and outputs will serve as guidance for the following two stages. Instead of feeding uninformed noise as input for stage three, IIE stage aims to establish an initialized input with expressive HrHSI-relevant information by resorting to a spectral mapping learning network, thus facilitating the extraction of prior information and further magnifying the potential of DIP for high-quality reconstruction. Last, we propose a dual U-shape network as a powerful regularizer to capture image statistics, in which two U-Nets are coupled together by cross-attention guidance (CAG) module to separately achieve spatial feature extraction and final image generation. The CAG module can incorporate abundant spatial information into the reconstruction process and hence guide the network toward a more plausible generation. Extensive experiments demonstrate the effectiveness of our proposed M2U-Net in terms of quantitative evaluation and visual quality. The code will be available at https://github.com/JiaxinLiCAS.

The adequate and finer spectral information in hyperspectral images (HSIs) are benefit for various downstream applications like smart agriculture and environmental monitoring. In HSI classification, dual-stream convolutional networks have gained much attention and have been widely used. In patch-based hyperspectral classification tasks, however, merely using center-labeled patches could lead to an increased unlabeled noise in the data. Moreover, in the application of dual-stream network structures, heterogeneity existed in both the data and feature semantic levels to capture more representative features. To tackle these challenges, we have devised a framework called cross-semantic heterogeneous modeling network (CreatingNet), which aligns more closely with the design principles of dual-stream networks by adjusting the input size. This framework introduces a distance metric attention mechanism (DMAM) based on spectral and spatial distances to strengthen the influence of the center pixel on the entire patch. Additionally, we present a fusion module named CrossViT, which combines features with diverse structures and characteristics, leveraging their complementarity. The proposed multiscale heterogeneous fusion module allows for more effective integration of spatial and spectral features in the images. Extensive experiments on four well-known HSI datasets (Indian Pines, Pavia University, Salinas, and Houston 2013) demonstrate the superior classification performance of the proposed CreatingNet to several state-of-the-art methods. The effectiveness of the proposed model is further validated through ablation studies.

The most cost-effective way to obtain a high spatial resolution hyperspectral image (HrHSI) is to fuse a low spatial resolution hyperspectral image (LrHSI) and corresponding high spatial resolution multispectral image (HrMSI). This article proposes a generalizable unsupervised deep fusion method based on spectral-spatial collaborative constraint to address LrHSI and HrMSI fusion task. First, in view of the limitations of the current spectral-spatial downsampled model, the group convolution enhancement (GCE) module is designed to eliminate the radiometric difference between the images to be fused. Second, to enhance the model’s feature extraction ability, this article introduces the design of the spatial, channel, and filter 3-D attention factor dynamic convolutional kernel (SCFConv). In order to verify the proposed method, we compared and evaluated our method with traditional methods and unsupervised deep learning methods using both simulated and real onboard data, respectively. In the absence of HrHSI validation images in real scenarios, we evaluate the performance of different fusion models through classification results. The experimental results demonstrate the effectiveness of the proposed model and the practical value of the fusion results (the onboard data produced by ours are available athttps://drive.google.com/drive/folders/1JLCCB6ld5R49HDLN5SsMISx1d0fuqRjO).

Single-image super-resolution (SISR) for high-resolution (HR) remote sensing image (RSI) acquisition is becoming increasingly valuable and important, and convolutional neural networks (CNNs) have produced considerable progress in this field. In RSIs, many similar geo-objects recur within the same scene, maintaining the same positions in both low resolution (LR) and HR. Based on this observation, we found that these similar geo-objects could be utilized to reconstruct texture details in LR by exploiting the consistent non-local relationships between these geo-objects in LR and HR, thereby improving the quality of SISR. Therefore, we propose a novel graph convolutional network (GCN) for SISR including a dynamic graph attention mechanism to learn the in-scale and cross-scale non-local features of RSIs. In scale, we propose a dynamic graph attention block (DGAB) that adaptively determines non-local patches upon the scene correlation derived from RSIs and further fuses patch-wise non-local information weighed by the attention scores of topological relationships and radiation characteristics in RSIs. Across different scales, we also introduce a dynamic graph attention mixing block (DGAMB) to upsample LR non-local information to HR non-local information. Most SISR methods have the upsampling blocks at the end of the network, ignoring feature extraction in high-dimensional space. To address this problem, DGAMB was designed as an upsampler in the middle of the model, enhancing the level of high-dimensional information extraction from the model. The experiments based on the WHU Building and UC Merced datasets show that our proposed method outperforms state-of-the-art methods. Our code is available athttps://github.com/WenjuanZhang-aircas/NSGCN.

Owing to the strong ability of feature extraction and representation, deep learning has exhibited powerful potential in the field of multispectral-aided hyperspectral super-resolution (MS-aided HS-SR). Though great strides have been made by existing methods, their superior performance mainly drives from large training datasets, hence failing to handle the real cases with limited samples. In this article, we propose an unsupervised method inspired by the theory of spectral mixing, which is based solely on one pair of HS-MS correspondence. Specifically, two coupled autoencoders are employed as the backbone of our network, aiming at deriving the latent abundance representations and corresponding endmembers of input HS-MS data. To enrich the feature representations and guide the network learning, we embed an interactive module into the encoder part to enhance the information transmission and design a degradation loss to constrain the target image. Experiments in Chikusei dataset demonstrate the effectiveness of our proposed method.

In recent years, fusion methods based on unsupervised deep learning have achieved impressive performance in the fusion of hyperspectral image (HSI) and multispectral image (MSI). However, there are still some limitations in the current research. Most existing fusion methods only apply to simulated data and need more verification on real data sets. To solve these issues, this paper designed an unsupervised dynamic convolutional neural network fusion model (UDCNN), which can adaptively learn the radiometric difference between HSI and MSI. This model achieves better performance on simulated data compared with related unsupervised deep learning methods, and achieves more accurate results on real data through classification-oriented application of the fusion results.

Fusing a low-resolution hyperspectral image (LrHSI) with an auxiliary high-resolution multispectral image (HrMSI) is a burgeoning technique to realize hyperspectral image super-resolution, in which learning-based methods have dominated the mainstream direction. However, the underutilization of degradation models and strong dependence on large-scale training triplets severely impedes their applicability and performance. Considering these issues, we reformulate the fusion task as a spectral mapping problem and hence propose an unsupervised model-guided coarse-to-fine fusion network. Specifically, degradation knowledge learning is first performed to fully excavate latent model information, which will serve as guidance for better mapping learning. Following that, a coarse-to-fine fusion network is constructed with a multi-scale attentional fusion module in the head and a coarse-to-fine structure in the tail. The former is deployed to achieve a more informative compression, and the latter is adopted to capture the spectral relationship, including a spectral degradation-guided subnetwork for group-by-group coarse reconstruction and a refinement subnetwork for inter-group correlation and dependencies. Finally, high-resolution HSI can be recovered via established spectral mapping. Extensive experiments on simulated and real datasets verify the superiority of our proposed method. The code is available at https://github.com/JiaxinLiCAS/UMC2FF_GRSL.

Recently, unmixing-based networks have shown significant potential in unsupervised multispectral-aided hyperspectral image super-resolution task (MS-aided HS-SR). Nevertheless, the representation ability of unsupervised networks and the design of loss functions still have not been fully explored, leaving large room for further improvement. To this end, we propose an enhanced unmixing-inspired unsupervised network with attention-embedded degradation learning, EU2ADL for short, to realize MS-aided HS-SR. First, two coupled autoencoders serve as the backbone of EU2ADL network to simultaneously decompose input modalities into abundances and corresponding endmembers, whose encoder part is composed of a spatial-spectral two-stream subnetwork for modality-salient representation learning and a parameter-shared one-stream subnetwork for modality-interacted representation enhancement. More importantly, a hybrid model-constrained loss containing a perceptual abundance term and a degradation-guided term is introduced to further eliminate the latent distortions. Since the hybrid loss is built on the degradation model, we additionally present an attention-embedded degradation learning network to adaptively estimate the unknown degradation parameters. Extensive experimental results on four datasets demonstrate the effectiveness of our proposed methods when compared with state-of-the-arts.

Hyperspectral image super-resolution can compensate for the incompleteness of single-sensor imaging and provide desirable products with both high spatial and spectral resolution. Among them, unmixing-inspired networks have drawn considerable attention owing to their straightforward unsupervised paradigm. However, most do not fully capture and utilize the multi-modal information due to their limited representation ability of constructed networks, hence leaving large room for further improvement. To this end, we propose an X-shaped interactive autoencoders network with cross-modality mutual learning between hyperspectral and multispectral data, XINet for short, to cope with this problem. Generally, it employs a coupled structure equipped with two autoencoders, aiming at deriving latent abundances and corresponding endmembers from input correspondence. Inside the network, a novel X-shaped interactive architecture is designed by coupling two disjointed U-Nets together via a parameter-shared strategy, which not only enables sufficient information flow between two modalities but also leads to informative spatial-spectral features. Considering the complementarity across each modality, a cross-modality mutual learning module is constructed to further transfer knowledge from one modality to another, allowing for better utilization of multi-modal features. Moreover, a joint self-supervised loss is proposed to effectively optimize our proposed XINet, enabling an unsupervised manner without external triplets supervision. Extensive experiments, including super-resolved results in four datasets, robustness analysis, and extension to other applications, are conducted, and the superiority of our method is demonstrated.

Hyperspectral images (HSIs) usually have finer spectral resolution but coarser spatial resolution than multispectral images (MSIs). To obtain a desired HSI with higher spatial resolution, great research attention has been paid to achieving hyperspectral super-resolution by fusing the observed HSI with an auxiliary MSI of the same scene. However, most of the existing HSI-MSI fusion methods rely either on prior knowledge of the degradation model or on sufficient training data, hindering their practicality and interpretability. In this letter, we propose a novel unsupervised HSI-MSI fusion network with the ability of degradation adaptive learning, namely, UDALN. Specifically, we propose three modules to straightly encode the spatial and spectral transformations across resolutions, i.e., SpaDnet, SpeUnet, and SpeDnet. Through an elaborately designed three-stage unsupervised training strategy, the estimated network parameters can exhibit clear physical meanings of degradation processes and therefore help guarantee a faithful reconstruction of the desired HSI. The experimental results on two widely used hyperspectral datasets demonstrate the effectiveness of our method in comparison to the state-of-the-art HSI-MSI fusion models. (Code available athttps://github.com/JiaxinLiCAS/UDALN_GRSL.)