Pan-sharpening aims to generate high-resolution multispectral images by integrating the spectral richness of low-resolution multispectral images with the spatial details of high-resolution panchromatic images. Although frequency-domain modeling shows great potential in this field, most existing methods are still limited to spatial-domain processing or fail to effectively capture the contextual interactions between frequency and spatial features. To address these issues, we propose a novel multi-scale frequency-spatial collaborative fusion approach. A Frequency-Spatial U-Net is introduced as the backbone network, in which frequency-spatial modeling blocks are embedded to progressively enhance the frequency-guided spatial contextual modeling capability across layers. To this end, we design a Dual Branch Frequency Attention module that adaptively enhances high- and low-frequency information. In addition, we introduce fine-mid-coarse resolution branches and devise a main-auxiliary multi-scale reconstruction loss to facilitate collaborative optimization. The effectiveness of the proposed model is validated through extensive experiments, demonstrating superior performance in both qualitative and quantitative evaluations. Moreover, our model achieves the fastest inference time among all compared methods, striking an excellent balance between accuracy and efficiency.

With the increasing need for intelligent and secure multimedia systems, infrared and visible image fusion (IVIF) has garnered a lot of attention due to its ability to overcome the limitations of a single sensor and integrate unique information from different modalities. However, it is common to overlook how the spatial frequency information of visible and infrared images differs. A less thorough feature extraction may result from many approaches’ inability to reconcile the extraction of both global and local information. To solve the aforementioned difficulties, we propose a dense multiscale fusion network DMNet. Through a dual-stream collaborative feature decoupling, the proposed network optimizes both the encoder–decoder network and the diffusion model to extract multimodal information more comprehensively. Specifically, the three-stage progressive encoder sequentially integrates dense transformer block (DTB) and dense invertible neural network block (DIB) to achieve global feature extraction and multimodal feature decoupling. Our proposed channel and spatial attention block (CSAB) selectively focuses on the important feature maps to better capture the critical information. Additionally, multiscale latent features are extracted by the diffusion module (DM) to enhance the representation of cross-modal latent features. As demonstrated by extensive experiments, DMNet outperforms representative state-of-the-art methods. Furthermore, we conduct sufficient ablation experiments to validate each module’s effectiveness, and we demonstrate that DMNet can enhance downstream infrared-visible object detection performance. Our fused results and code will be accessible athttps://github.com/Pancy9476/DMNet.

To obtain fused images with rich information, visible and infrared images are combined. Most current fusion techniques provide decent results. However, they have shortcomings in extracting information of the source images. This limitation prevents the fused images from adequately considering thermal radiation regions and texture details. As a result, the detailed texture information of the source visible image in the final fusion image is much more than the thermal target information of the source infrared image, or vice versa. Since features at a single scale fail to adequately capture the spatial details of complex scenes, a multi-scale attention network is used to extract the deep feature information of source images. For latent variable issues, the Expectation Maximization (EM) technique can yield maximum likelihood estimates. This not only stabilizes the training of the Generative Adversarial Network (GAN) but also aids in addressing the issue of labels lacking in the fusion of visible and infrared images. Although the EM algorithm framework can greatly enhance the training stability of GAN models, the improvement in fusion quality is not large. Therefore, a diffusion model is introduced into the generator to capture the potential joint structure information between infrared and visible images. Massive experiments show that Dif-GAN outperforms the state-of-the-art.