Currently, most Hyperspectral (HS) pansharpening methods have two problems, namely the lack of consideration the spatial variations of HS images and inaccurate feature reconstruction in multi-channel complex mapping relationships, leading to spectral and spatial distortions in the fusion results. To address these issues, we propose a dynamic network based on feature modulation and probability mask (FMPM-DNet) for HS pansharpening, including two stages of spectral-spatial feature modulation and feature reconstruction. In the first stage, to increase the feature representation ability of the model, a wave function is defined based on complex transformation to convert spatial features into wave-like features. On this basis, considering the spatial variations of HS images, a dynamic feature modulation unit (DFMU) is constructed to achieve adaptive modulation and coarse fusion of features by dynamically generating spectral-spatial correction matrix. In the second stage, a feature probability mask unit (FPMU) is designed to realize global feature embedding at different depths and local feature embedding at the same depth to obtain refined fused features. Extensive experiments on three widely used datasets demonstrate that the proposed FMPM-Net achieves significant improvements in both spatial and spectral quality metrics compared to some state-of-the-art (SOTA) methods.

Fusion of a high-spatial-resolution multispectral image (MSI) and a low-spatial-resolution hyperspectral image (HSI) aims to generate a high-spatial-resolution HSI (HR-HSI). Most fusion methods use simple upsampling techniques to increase the resolution of HSI without guidance, which can introduce unwanted artifacts and lead to spectral distortion. Additionally, they face challenges in generalization and robustness. To address these challenges, this paper introduces a cross-modal fusion network for MSI and HSI, named CSGAV, which is built on similarity-guided graph attention (SGA) and a variational autoencoder-Transformer (VAET). Specifically, we develop a similarity measure algorithm to compute the similarity degree between the source images and construct an SGA module to mitigate modal differences, producing precise upsampled outputs. Moreover, we present an adaptive weighted Transformer, with the weights guided by a variational autoencoder, thereby enhancing the generalization and robustness of the model. The SGA and VAET are integrated in the cross-modal interactive architecture to achieve the final HR-HSI image. Experimental results conducted on four public datasets show that CSGAV is superior compared to existing state-of-the-art fusion methods both in fusion performance and generalization. The code of this work is available at https://github.com/yotick/CSGAV.

In this letter, to better supplement the advantages of features at different levels and improve the feature extraction ability of the network, a novel multi-level feature interaction transformer network (MFITN) is proposed for pansharpening, aiming to fuse multispectral (MS) and panchromatic (PAN) images. In MFITN, a multi-level feature interaction transformer encoding module is designed to extract and correct global multi-level features by considering the modality difference between source images. These features are then fused using the proposed multi-level feature mixing (MFM) operation, which enables features to fuse interactively to obtain richer information. Furthermore, the global features are fed into a CNN-based local decoding module to better reconstruct high-spatial-resolution multispectral (HRMS) images. Additionally, based on the spatial consistency between MS and PAN images, a band compression loss is defined to improve the fidelity of fused images. Numerous simulated and real experiments demonstrate that the proposed method has the optimal performance compared to state-of-the-art methods. Specifically, the proposed method improves the SAM metric by 7.89% and 6.41% compared to the second-best comparison approach on Pléiades and WorldView-3, respectively.

Pansharpening technology aims to extract spatial information from a panchromatic (PAN) image and integrate it into a multispectral (MS) image to generate a high-spatial resolution MS image. To overcome the problem of spatial and spectral distortion of traditional methods, this letter presents a pansharpening method based on fuzzy logic and edge activity. To obtain more accurate spatial details of source images, details extracted using the component substitution and multiresolution analysis methods are fused via the proposed fuzzy logic algorithm. To preserve the edges of the fusion result, the edge maps of the source images are detected and fused based on the edge activity. The optimized detail maps are obtained by multiplying the fused details and edge maps, which are then injected into the upsampled MS image to obtain the final pansharpened image. Reduced- and full-scale experimental results on the Pléiades and IKONOS datasets demonstrate the effectiveness of our method compared with state-of-the-art pansharpening algorithms. Specifically, the proposed method improves the ERGAS metrics by 9.0% and 11.1% compared to the second-best comparison approach on Pléiades and IKONOS, respectively.

Pansharpening aims to obtain a high-spatial-resolution multispectral (MS) image by fusing a lower-spatial resolution MS image with a high-spatial-resolution panchromatic (PAN) image. Currently, the results obtained by most pansharpening methods still suffer from spatial and spectral distortion issues. The diffusion model has shown outstanding performance in various image-processing tasks. However, maintaining the full image size throughout the diffusion process imposes a large computational burden, and the simultaneous use of PAN and MS images acquired by different sensors as a condition for guiding noise prediction leads to spatial and spectral distortions. To solve these problems, a variable-scale diffusion model (VSDM) based on dynamic condition guidance for pansharpening is proposed, which achieves better fusion performance by improving the diffusion manner of the diffusion model and injecting dynamic conditions to guide the reverse process. In VSDM, a variable-scale diffusion manner (VSDMN) is designed to reduce the computational complexity of the model by reducing the size of the image in the diffusion process. A condition generator (CG) is constructed to generate dynamic conditions using the features learned from the PAN and upsampled MS images. In CG, a cross-attention dynamic convolution is built to extract features from the PAN image by designing a spatial and spectral attention mechanism, which can improve the spatial and spectral consistency in the dynamic condition. Extensive experiments validate the effectiveness of the proposed VSDM against other state-of-the-art (SOTA) pansharpening methods in both quantitative and qualitative assessments. The source code will be released athttps://github.com/MELiMZ/VSDM.

Pansharpening is to fuse a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LRMS) image to obtain an enhanced LRMS image with high spectral and spatial resolution. The current Transformer-based pansharpening methods neglect the interaction between the extracted long- and short-range features, resulting in spectral and spatial distortion in the fusion results. To address this issue, a novel cross Transformer and convolutional neural network (CNN) for pansharpening (CTCP) is proposed to achieve better fusion results by designing a cross mechanism, which can enhance the interaction between long- and short-range features. First, a dual branch feature extraction module (DBFEM) is constructed to extract the features from the LRMS and PAN images, respectively, reducing the aliasing of the two image features. In the DBFEM, to improve the feature representation ability of the network, a cross long-short-range feature module (CLSFM) is designed by combining the feature learning capabilities of Transformer and CNN via the cross mechanism, which achieves the integration of long-short-range features. Then, to improve the ability of spectral feature representation, a spectral feature enhancement fusion module (SFEFM) based on a frequency channel attention is constructed to realize feature fusion. Finally, the shallow features from the PAN image are reused to provide detail features, which are integrated with the fused features to obtain the final pansharpened results. To the best of our knowledge, this is the first attempt to introduce the cross mechanism between Transformer and CNN in pansharpening field. Numerous experiments show that our CTCP outperforms some state-of-the-art (SOTA) approaches both subjectively and objectively. The source code will be released at https://github.com/zhsu99/CTCP.

Pansharpening is to fuse high-resolution panchromatic (PAN) images with low-resolution multispectral (LR-MS) images to generate high-resolution multispectral (HR-MS) images. Most of the deep learning-based pansharpening methods did not consider the inconsistency of the PAN and LR-MS images and used simple concatenation to fuse the source images, which may cause spectral and spatial distortion in the fused results. To address this problem, a multi-scale spatial-spectral attention guided fusion network for pansharpening is proposed. First, the spatial features from the PAN image and spectral features from the LR-MS image are independently extracted to obtain the shallow features. Then, a spatial-spectral attention feature fusion module (SAFFM) is constructed to guide the reconstruction of spatial-spectral features by generating a guidance map to achieve the fusion of reconstructed features at different scales. In SAFFM, the guidance map is designed to ensure the spatial-spectral consistency of the reconstructed features. Finally, considering the difference between multiply scale features, a multi-level feature integration scheme is proposed to progressively achieve fusion of multi-scale features from different SAFFMs. Extensive experiments validate the effectiveness of the proposed network against other state-of-the-art (SOTA) pansharpening methods in both quantitative and qualitative assessments. The source code will be released at https://github.com/MELiMZ/ssaff.

Deep learning (DL)-based pansharpening methods have shown great advantages in extracting spectral–spatial features from multispectral (MS) and panchromatic (PAN) images compared with traditional methods. However, most DL-based methods ignore the local inner connection between the source images and the high-resolution MS (HRMS) image, which cannot fully extract spectral–spatial information and attempt to improve the quality of fusion by increasing the complexity of the network. To solve these problems, a lightweight network based on adaptive weighted feature learning network (AWFLN) is proposed for pansharpening. Specifically, a novel detail extraction model is first built by exploring the local relationship between HRMS and source images, thereby improving the accuracy of details and the interpretability of the network. Guided by this model, we then design a residual multiple receptive-field structure to fully extract spectral–spatial features of source images. In this structure, an adaptive feature learning block based on spectral–spatial interleaving attention is proposed to adaptively learn the weights of features and improve the accuracy of the extracted details. Finally, the pansharpened result is obtained by a detail injection model in AWFLN. Numerous experiments are carried out to validate the effectiveness of the proposed method. Compared to traditional and state-of-the-art methods, AWFLN performs the best both subjectively and objectively, with high efficiency. The code is available athttps://github.com/yotick/AWFLN.

Pansharpening aims at fusing a multispectral (MS) image and panchromatic (PAN) image to obtain a high spatial resolution multispectral (HRMS) image. To obtain accurate details and reduce spectral distortion, this letter proposes an efficient pansharpening approach based on texture correction (TC) and detail refinement. First, a TC model is constructed based on spatial and spectral fidelity constraints to obtain a texture image that is highly correlated with the MS image. Second, a detail acquisition model is proposed by the consecutive parameter regression to adaptively refine the extracted details. Finally, the extracted details are injected into the up-sampled MS (UPMS) image to obtain the fused HRMS image. Experimental results demonstrate that the proposed method can obtain the high-quality results with high efficiency.

Pansharpening is a technology involving information integration and processing in remote sensing imagery. It is applied to generate a high-resolution multispectral (HRMS) image through an effective fusion of a low spatial resolution multispectral image and a panchromatic (PAN) image. In this paper, we propose an end-to-end multi-scale and multi-distillation dilated network (MMDN) for pansharpening. In MMDN, to extract more abundant spatial details from source images, a clique structure-based multi-scale dilated block (CSMDB) is presented. The clique structure in CSMDB can fully transfer the information between feature maps obtained by the multi-scale dilated convolutional filters. Then, a multi-distillation residual information block (MRIB) is constructed to help the network capture the spatial structure of different scales in MS and PAN images. Finally, to reuse and supplement the feature information, a feature embedding strategy is designed by feeding the sum result of the output of cascaded CSMDBs and the shallow features to each MRIB. Experimental results verify that the proposed MMDN outperforms other compared state-of-the-art approaches in terms of objective and subjective evaluations.

Deep-learning-based pansharpening methods have achieved remarkable results due to their powerful feature representation ability. However, the existing deep-learning-based pansharpening methods not only lack information exchange and sharing between features of different resolutions but also cannot effectively use the residual information at different levels. These disadvantages may lead to the loss of spatial information and spectral information in the pansharpened image. To address the above problems, we propose a novel dual-stream convolutional neural network with residual information enhancement (DSCNN-RIE) for pansharpening. The proposed network is mainly composed of a set of dual-stream information complementation blocks (DSICBs), which can extract various spatial details at two different resolutions using convolutional filters of various sizes simultaneously, and can transfer complementary information effectively between two different resolutions. Furthermore, to improve the learning ability of the network and enhance the feature extraction, an RIE strategy is presented to stack different levels of residuals into the outputs of cascaded DSICBs. The final pansharpened image is obtained by integrating the extracted features using the shallow feature information of the source images. Experimental results on three datasets demonstrate that DSCNN-RIE outperforms ten other state-of-the-art pansharpening methods in both subjective and objective image-quality evaluations.

Pansharpening is used to fuse a panchromatic (PAN) image with a multispectral (MS) image to obtain a high-spatial-resolution multispectral (HRMS) image. Traditional pansharpening methods face difficulties in obtaining accurate details and have low computational efficiency. In this study, a unified pansharpening model based on the band-adaptive gradient and detail correction is proposed. First, a spectral fidelity constraint is designed by keeping each band of the HRMS image consistent with that of the MS image. Then, a band-adaptive gradient correction model is constructed by exploring the gradient relationship between a PAN image and each band of the MS image, so as to adaptively obtain an accurate spatial structure for the estimated HRMS image. To refine the spatial details, a detail correction constraint is defined based on the parameter transfer by designing a reduced-scale parameter acquisition model. Finally, a unified model is constructed based on the gradient and detail corrections, which is then solved by an alternating direction multiplier method. Both reduced-scale and full-scale experiments are conducted on several datasets. Compared with state-of-the-art pansharpening methods, the proposed method can achieve the best results in terms of fusion quality and has high efficiency. Specifically, our method improves the SAM and ERGAS metrics by 17.6% and 21.2% respectively compared to the traditional approach with the best average values, and improves these two metrics by 4.3% and 10.3% respectively compared to the learning-based approach with the best average values.

Pansharpening is to fuse the existing low spatial resolution multi-spectral (MS) image with high spatial resolution panchromatic (PAN) image, so as to obtain high spatial resolution MS (HRMS) image. An efficient pansharpening model based on conditional random fields (CRFs) is proposed in this paper. In the model, a state feature function is designed to force the blurred HRMS image in accordance with the upsampled MS (UPMS) image to keep the spectral fidelity. Meanwhile, a transition feature function is defined to keep the sharpness of fused image. Besides, a new Gaussian filter acquisition algorithm is proposed to effectively satisfy the blur function in the model. To improve the efficiency of algorithm, a new initialization method based on fitting normal distribution is presented. Experiments are conducted on both reduced-scale images and full-scale images. Compared with some classical and state-of-art pansharpening methods, the proposed method achieves the best results in terms of fusion quality and efficiency.

Remote sensing image fusion is to fuse low spatial resolution multispectral (MS) images with high spatial resolution panchromatic (PAN) images to get high spatial resolution multispectral images. The component substitute (CS)-based methods are popular approaches for their high efficiency and high spatial resolution. However, they may produce spectral distortion, especially when there are large radiometric differences between PAN images and MS images. For tackling this problem, a new framework based on the CS model with fuzzy logic and salience measure is proposed. In order to get details that are highly relevant to MS image, a novel fuzzy logic rule based on the global salience measure is designed to fuse the details extracted from both PAN image and MS image. Furthermore, to better preserve the edges of the fused image, a new edge-preserving algorithm is defined to fuse the edges from the PAN image and MS image according to the local salience measure. A series of experiments are conducted and analyzed on both simulated images and real images from the data sets of four satellites to illustrate the effectiveness of the proposed method. Compared with some state-of-the-art methods, our method performs the best in both objective and subjective evaluations. Besides, our method has a low computational cost and is suitable for practical application.