Satellite-based hyperspectral (HS) imagery holds great potential in remote sensing applications due to its fine spectral resolution. However, the low spatial resolution limits its practical utility. Combining ancillary high resolution panchromatic (PAN) or multispectral (MS) images has become a common practice to improve the spatial quality of HS images. Most present approaches, however, are based on dual-sensor fusion (e.g., MS-HS or PAN-HS), which generally falls short of comprehensively integrating their complementary spatial and spectral information of PAN, MS, and HS images. Meanwhile, existing few integrated fusion methods suffer from two key limitations: modality mismatch due to inconsistent spatial-spectral characteristics among PAN, MS, and HS data, and shallow and redundant cross-modal coupling caused by inadequate modeling of inter-modal relationships. In this paper, we propose a progressive spatial-spectral interactive network (PSSNet) for the integrated fusion of panchromatic, multispectral, and hyperspectral images. Specifically, a context-aware fusion block is introduced to extract and enhance contextual spatial and spectral information across different modalities. To ensure an effective integration of spatial and spectral details, the entire network is structured progressively, allowing for a smooth transition and fusion of features from HS, MS, and PAN images. Additionally, a spatial-spectral feature recombination module is designed to dynamically adjust the contribution of spectral features at various levels. This module, in combination with a spatial enhancement component, facilitates the optimal fusion of spatial and spectral information by enhancing their interactions. Extensive experiments on simulated and real datasets, both qualitatively and quantitatively, demonstrate the superiority of PSSNet compared to other state-of-the-art methods.

The integrated fusion of the high-spatial-resolution (HR) panchromatic image (PAN), the relative “moderate”-spatial-resolution (MR) multispectral image (MSI), and the low-spatial-resolution (LR) hyperspectral image (HSI), to generate the optimal HR HS fused image, is promising but challenging. On the one hand, existing mainstream fusion models mostly focus on the “pairwise fusion” between HR PAN, MR MSI, and LR HSI, which cannot sufficiently integrate their complementary spatial and spectral advantages. On the other hand, one of the few integrated fusion methods roughly introduced the MR MSI as a simple intermediate medium; however, the role of MSIs as a spatial and spectral “bridge” between HR PAN and LR HSI, generally characterized by significant scale differences, remains largely unexplored. To solve these problems, we proposed an integrated PAN–MSI–HSI fusion method from the perspective of MSIs, by comprehensively considering the scale difference among the multisource observations. In the proposed method, a spatial–spectral feature transfer network was designed by comprehensively exploring the spatial–spectral variations and connections among the HR PAN, MR MSI, and LR HSI. Then, a spatial–spectral joint reconstruction module was constructed to reconstruct the HR HSI with optimal spatial and spectral fidelity. Experiments were conducted on simulated and real datasets from qualitative and quantitative aspects. The experimental results demonstrated the competitive effectiveness over other state-of-the-art methods.

The integration of hyperspectral (HS) imagery and light detection and ranging (LiDAR) data for land cover classification has emerged as a prominent research focus. Despite the satisfactory classification accuracies achieved by existing methodologies, several unaddressed issues that remain warrant consideration. First, current approaches overlook the pronounced spectral and spatial heterogeneities in remote sensing (RS) images designated for multiclassification tasks, limiting the performance of classification models. Moreover, most existing studies amalgamate elevation features with other characteristics through simple addition and interaction operations, and they do not delve deeply into exploiting elevation height information, leading to an imbalance in the representation of elevation height. In light of the aforementioned issues, this article introduces a spatial-spectral heterogeneity-aware network (S2HANet) for the joint classification of HS and LiDAR data. Specifically, a shared spectral correction module (SSCM) is designed in the spectral branch to preliminarily alleviate the problem of large intraclass variance, followed by the use of a contrastive learning framework to enhance the intraclass compactness and interclass separability of spectral features. A multichannel signed distance discrimination module (MCSDDM) is developed to learn the distance relationships between intra- and interclass pixels and boundaries, and using prior boundary information to improve spatial boundary information. In addition, an elevation boost module (EBM) and an elevation injection module (EIM) are meticulously designed to phase-in elevation height information, further enhancing the utilization of elevation data and better facilitating the fusion of the two modalities. The proposed S2HANet has demonstrated exceptional classification performance across three opening benchmark datasets.

Abstract Infrared and visible image fusion involves merging the advantages of infrared and visible images to generate a composite image that encompasses thermal radiation as well as intricate texture details. Infrared and visible image fusion has garnered increasing attention, with numerous fusion methods proposed. However, how to fairly perceive the performance of fused image remains a contentious topic. This paper is dedicated to solving this problem from two perspectives (e.g., subjective and objective aspects). Firstly, an infrared and visible fusion image quality assessment dataset was constructed, including 60 pairs of infrared and visible images captured in various scenes, along with 540 fusion images with different types and degrees of distortions. Additionally, a subjective evaluation dataset of 16,200 subjective scores by 30 participants was further provided for the fused image. Secondly, to overcome the challenging assessment for infrared and visible fusion images without a real reference image, an interesting multi‐domain pseudo‐reference image quality assessment model (MPIQAM) is proposed, by comprehensively considering the thermal radiation information distortion, texture information distortion, and overall naturalness of the fused image. The proposed MPIQAM was compared with 18 mainstream objective metrics, and the experimental findings showcased a commendable level of competitiveness.

The integration of hyperspectral (HS) imagery and light detection and ranging (LiDAR) for land cover classification has become a significant research topic. Numerous existing methods aim to interactively fuse the complementary features of HS and LiDAR to enhance the classification accuracy. However, most existing studies overlook the fact that spectral, spatial, and elevation features of HS and LiDAR possess significant discriminative information for specific categories. The rough and simple interacting or stacking these features may hinder the effective expression of this significant discriminative information. Moreover, existing approaches neglect the shared spatial characteristics between HS and LiDAR. In this article, an uncertain category-aware fusion network (UCAFNet) is proposed to tackle the above challenges. Specifically, we proposed an uncertain category-aware fusion strategy (UCAFS) that dynamically weights the spectral, spatial, and elevation branches based on their respective capabilities in identifying different categories to achieve targeted information aggregation. Moreover, we introduce the spatial information purification module (SIPM) and adaptive weighted fusion module (AWFM), to extract and enhance shared spatial features from HS and LiDAR for effective integration. The experimental results on three public benchmark datasets demonstrate the superior performance of the proposed UCAFNet.

Spatiotemporal fusion has been attracting increasing attention in remote sensing applications, such as environmental monitoring and land cover change detection, due to its excellent ability to obtain high spatial and temporal resolution images. The land cover change has always been a great challenge in spatiotemporal fusion. Although most spatiotemporal fusion methods have demonstrated satisfactory performance in addressing phenological changes, the performance in terms of abrupt land cover type changes, such as floods or mudslides, falls short. To alleviate this issue, we propose a change information guided spatiotemporal fusion (CIG-STF) method. The proposed CIG-STF integrates change detection and spatiotemporal fusion in a unified framework, by taking advantage of change detection in capturing land cover changes to assist spatiotemporal fusion. Specifically, the CIG-STF comprises three modules: multiscale dilated feature extractor module (MDFE), spatiotemporal fusion-change detection integrated module (STF-CD), and reconstruction module. The MDFE employs multiscale dilated convolutions to comprehensively extract features, to prevent crucial information loss by increasing the convolutional receptive field. In the STF-CD, a change detection module on attention strategy is integrated into the spatiotemporal fusion task, by excavating land cover changes to further enhance the fusion performance. In addition, we design a dynamic decay loss function to further leverage change information, ensuring the accuracy of both change information and prediction results. The experiments were verified on the publicly available LGC and Daxing datasets with manual change labels. The experimental results demonstrate the superior performance of the proposed CIG-STF in both phenological variations and land cover type changes.

Optical remote sensing images have the advantages in clear visual characteristics and strong interpretability. Unfortunately, cloud coverage limits the quality and availability of optical images in practical applications. In contrast, Synthetic Aperture Radar (SAR) images provide all-day and all-weather imaging, which can serve as effective auxiliary information for cloud removal. Existing cloud removal methods are difficult to obtain high-fidelity cloud-free results due to the insufficient spectral and spatial information exploration in the multitemporal SAR and optical images. In this paper, we propose a multi-discriminator supervision-based dual-stream interactive network (MDS-DIN) for cloud removal. Specifically, we first design a dual-stream interactive learning module to take full advantage of the complementary information between multitemporal SAR and optical images. Moreover, we specially design an adaptive weight fusion module to adaptively allocate fusion weights to the dual-stream results by considering the discriminative features in spectral and spatial levels. In addition, multi-discriminator is employed to jointly optimize overall networks for high-fidelity cloud removal. Experiments on simulated and real data sets demonstrate the competitive performance of our proposed method.

Multispectral (MS)-hyperspectral (HS) image fusion, which aims to enhance the spatial resolution of low spatial resolution HS images with a high spatial resolution MS has provided a wide range of applications in remote sensing. However, relatively long revisit cycles of HS satellites and irresistible weather factors cause the acquisition of HS and MS images at the same time difficult. Most of the existing approaches neglect the temporal difference between MS and HS images, and perform weakness in the challenging case with diverse temporal difference spans. In this paper, we propose a novel image fusion strategy with embedding a stage of feature matching before interaction. On the one hand, we explore the role of spectral correlation modeling between HS and MS images, which accounts for the utilization of available spatial information from MS images. On the other hand, we design a feature aggregation module to fully exploit the nonlinear gaps and dependencies of heterogeneous data and utilize adaptive gains to realize complementary information projection and fusion. We build Dongying (DY) and Yellow River Estuary (YRE) remote sensing datasets based on Sentinel-2 and ZiYuan(ZY)-1 02D satellites with diverse temporal difference spans. The extensive experiments demonstrate that our method is robust to the span of temporal difference and shows superior performance over the existing methods visually and quantitatively.

Spatio-temporal-spectral fusion aims to produce high spatio-temporal-spectral resolution images by integrating the complementary spatial, temporal, and spectral advantages of multi-source remote sensing images. However, on one hand, existing spatio-temporal-spectral fusion methods are insufficient to exploit the inherent complex nonlinear spatial, temporal, and spectral relationship among multisource and multitemporal observations. On the other hand, since the unavailability of real high spatio-temporal-spectral resolution images, it is difficult to adopt deep learning methods with supervised training. In this paper, we propose an effective Unsupervised Spatio-Temporal-Spectral Fusion Model (USTSFM) with dual-task interactive learning to alleviate these problems. The proposed USTSFM has two branches: the Spatio-Temporal-Spectral Mapping (STSM) branch is to describe the temporal relationship, and the Spectral Super Resolution (SSR) branch is to model the spectral relationship. Moreover, the spatial-spectral interaction compensation block is designed to make the two branches compensate and benefited from each other. This intrinsically related and mutually facilitated strategy allows the USTSFM to sufficiently exploit the inherent spatial, temporal, and spectral relationship. In addition, a shared reconstruction module is meticulously designed for the two tasks, which not only reduces the parameters but also allows the supervised task to guide the convergence of the unsupervised task, boosting the stability of unsupervised training. The qualitative and quantitative results demonstrated the proposed USTSFM has richer spatial details and more accurate predictions than the other state-of-the-art methods.

Spatio-temporal fusion can generate time-series images with high spatial resolution, and it is highly desirable in various applications, especially in monitoring fine dynamic changes of surface features on remote sensing images. Currently, most spatio-temporal fusion methods predict the target fine image by employing the auxiliary fine images on neighboring phases; however, they are generally limited in abrupt land cover changes between the target and the neighboring auxiliary images. In this paper, we propose a novel Detail Injection-based Spatio-Temporal Fusion (DISTF) model to alleviate this problem, by exploring the inherent relationship between the spatio-temporal fusion and spatio-spectral fusion. The proposed DISTF consists of three modules: a Three-branch Detail Injection (TDI) module, a Fine Detail Prediction (FDP) module, and a reconstruction module. The interpretable TDI module is inspired by spatio-spectral fusion, aiming to inject the non-changed detail information extracted from the neighboring fine images into the target coarse image, which can preserve the abrupt change information captured in the target coarse image. The FDP module is designed to further integrate the correlated information from the outputs of TDI and refine the spatial-spectral information to boost the fusion accuracy. Finally, the reconstruction module and the hybrid loss function are designed to more effective reconstruct the high-quality target fine image. The qualitative and quantitative experiment results on two datasets with different types of changes demonstrated that the proposed DISTF method achieves richer spatial detail and more accurate prediction than the eight existing methods.

Change detection on heterogeneous optical and synthetic aperture radar (SAR) images is soaring and plays a crucial role in monitoring land cover changes, such as disaster emergencies and natural resource monitoring. This is commonly recognized as a promising but challenging work due to the intrinsic differences in imaging mechanisms between the optical and SAR images. Recently, deep learning-based change detection methods based on two-step processing have attracted attention, i.e., first image translation between optical and SAR images to alleviate their modality differences and then change detection based on the translated images. However, image translation itself is a trouble task for the heterogeneous optical and SAR images. The unreliable image translation results further limit the accuracy of change detection. In this paper, to mitigate this problem, we propose a change detection model on domain adaptation by novelty integrating change detection and image reconstruction into a unified framework. Specifically, we first transform the optical and SAR images into an intermediate common domain for comparison. Moreover, cross reconstruction for optical and SAR images is designed to maintain the characteristics of the images and improve the performance of domain adaptation. In addition, a feedback guidance mechanism is circumspectly designed to co-optimize change detection and image reconstruction tasks. Extensive experiments were conducted on four publicly available datasets, the results demonstrate the effectiveness of our proposed method.

The spaceborne hyperspectral (HS) imagery with fine spectral information has broad application aspects; however, the low spatial resolution has limited the potential application values. Over the past few decades, a general strategy to improve the spatial resolution of the HS is to fuse the low spatial resolution (LR) HS with an auxiliary moderate spatial resolution (MR) multispectral (MS) or a high spatial resolution (HR) panchromatic (PAN) image. However, most of the existing methods mainly focus on two-sensor fusion with the LR HS and MR MS images (i.e., MS-HS fusion) or the LR HS and HR PAN images (i.e., the PAN-HS fusion). How to comprehensively combine the complementary spatial and spectral advantages of the LR HS, MR MS, and HR PAN observations, to obtain the optimal high-fidelity HR HS image is interesting and challenging. In this paper, we propose a multi-scale grouping feedback embedded integrated fusion network (MGFEI-Net) for the LR HS, MR MS, and HR PAN images. Specifically, an attention-based hybrid-scale integrated module is designed by considering the spatial scale diversity of the HR PAN, MR MS, and LR HS images. Moreover, a multi-scale grouping feedback embedded module with a top-to-bottom manner is proposed to capture more usual spatial-spectral features. Experiments were performed on the simulated and real datasets. Moreover, the robustness of the proposed PAN-MS-HS fusion under different large spatial resolution ratios (such as 8, 16, 32, 64) was analyzed. The experimental results demonstrated the competitive performance of the proposed method.

Zi Yuan (ZY)-1 02D satellite simultaneously provides the low spatial resolution (LR) and narrow swath-width hyperspectral (HS) image, the moderate spatial resolution (MR) multispectral (MS) image with a wider swath width, and the high spatial resolution (HR) panchromatic (PAN) image with the same wide swath width to the MR MS. How to comprehensively integrate their complementary advantages to obtain the wide swath-width and high-fidelity HR HS image is interesting but challenging. In this paper, we propose an integrated fusion method for the HR PAN, MR MS, and LR HS images with different swath widths, to generate the optimal wide swath-width HR HS image. The proposed method is based on the encoder-decoder learning framework. In the proposed fusion framework, a novel multi-branch encoder structure with an enhanced HS-encoder module and the multilevel spatial-spectral aggregation block is designed, by considering the difference in the spatial and spectral resolution among the multi-sensor images. The experiments on synthetic and real datasets from both qualitative and quantitative aspects demonstrated the competitive performance of the proposed method.

Spatio-temporal fusion aims to integrate multisource remote sensing images with complementary high spatial and temporal resolutions, so as to obtain time-series high spatial resolution fused images. Currently, deep learning (DL)-based spatio-temporal fusion methods have received broad attention. However, on one hand, most of the existing DL-based methods train the model in a band-by-band manner, ignoring the correlations among bands. On the other hand, the general coarse spatio-temporal changes in low spatial resolution images (e.g., MODIS) calculated at the pixel domain cannot completely cover the fine spatio-temporal changes in high spatial resolution images (e.g., Landsat), due to complex surface features and the general large spatial resolution ratio between fine and coarse images. Besides, the existing DL-based spatio-temporal fusion methods are insufficient in exploring multiscale information by only stacking convolutional kernels with different sizes. To alleviate the above challenges, we propose a progressive spatio-temporal attention fusion model in a multiband training manner based on generative adversarial network (PSTAF-GAN). Specifically, we design a flexible multiscale feature extraction architecture to extract multiscale feature hierarchies. Then, spatio-temporal changes are calculated on the feature domain in different feature hierarchies. Besides, a spatio-temporal attention fusion architecture is proposed to fuse the spatio-temporal changes and ground details in a coarse-to-fine manner, which can explore multiscale information more sufficient and gradually recover the target image. The results of quantitative and qualitative experiments on two publicly available benchmark datasets show that the proposed PSTAF-GAN can achieve the best performance compared with the state-of-the-art methods.

Spatio-temporal fusion by combining the complementary spatial and temporal advantages of multi-source remote sensing images to obtain time-series high spatial resolution images is highly desirable in monitoring surface dynamics. Currently, deep learning (DL)-based fusion methods have received extensive attention. However, existing DL-based spatio-temporal fusion methods are generally limited in fusing the images with land cover changes. In this paper, we propose a spatio-temporal-spectral collaborative learning framework for spatio-temporal fusion to alleviate this problem. Specifically, the proposed method integrates the convolutional neural network and recurrent neural network into a unified framework, consisting of three sub-networks: multi-scale siamese convolutional neural network, multi-layer convolutional recurrent neural network, and adaptive weighting fusion network. The multi-scale siamese convolutional neural network has a flexible weight-sharing network to extract multi-scale spatial-spectral features from multi-source remote sensing images. The multi-layer convolutional recurrent neural network is constructed on the convolutional long-short term memory units to comprehensively learn the land cover changes by spatial, spectral, and temporal joint features. The adaptive weighting fusion network with a spatio-temporal-spectral change loss is proposed to further improve the interpretability and robustness. The experiments were performed on the publicly available benchmark datasets featured by phenology and land cover type changes, respectively. The experimental results demonstrated the competitive performance of the proposed method than other state-of-the-art fusion methods.