Multimodal Image Fusion (MMIF) aims to synthesize complementary information from different modalities to generate comprehensive fused images, thereby facilitating downstream applications. Existing methods typically employ deep neural networks to directly construct high-dimensional image-to-image mappings, which is highly challenging, struggling to extract generalizable patterns for various fusion scenarios. Inspired by meta learning, we propose a learning-to-optimize fusion framework, named LTOFusion, which formulates image fusion as a trajectory optimization problem, decoupling the complicated fusion problem into multistage subproblems. Subsequently, a restricted state transition function based on flow matching is designed to compress the prediction space and lead the network to build an image-to-flow mapping and fine-tune the current fusion state. To facilitate model training, we collect intermediate fusion states and utilize a memory-replay strategy, further enhancing the sample diversity and model robustness. In addition, a hybrid loss with respect to intensity, gradient, structure, and local normalized cross-correlation is designed to improve image details and reduce potential artifacts for fusion results. Experimental results demonstrate that the proposed method achieves the state-of-the-art performance across multiple fusion tasks and downstream applications without requiring fine-tuning. The code is available at https://github.com/HeDan-11/LTOFusion.

Multimodal medical image fusion aims to integrate complementary information from different modalities to support clinical diagnosis and treatment. Although deep learning has significantly advanced this field, existing methods often overlook the differences between various fusion scenarios, making a single network inadequate for diverse fusion requirements. Therefore, we propose a novel scenario-adaptive fusion network. The network employs a two-stage training process. In the first stage, an autoencoder is trained for multiscale feature extraction and image reconstruction. In the second stage, the autoencoder parameters are frozen, and a Fusion Layer is trained to achieve multimodal feature integration. The Fusion Layer consists of a Scenario-Specific Fusion Module and a Scenario-General Fusion Module. The former uses a mixture-of-experts model to customize fusion strategies for different scenarios to optimize the fusion process. The latter employs a dual-path fusion structure based on standard convolution and deformable convolution gating mechanisms to achieve general feature fusion across multi-scenario. Compared to eleven state-of-the-art methods, our method demonstrates superior information integration and visual consistency, offering a flexible and efficient solution for various fusion scenarios.

Medical image fusion technology provides professionals with more detailed and precise diagnostic information. This paper introduces a new efficient CT and MRI fusion network, CLGFusion, based on a contrastive learning-guided network. CLGFusion includes two encoding branches at the feature encoding stage, enabling them to interact and learn from each other. The approach begins with training a single-view encoder to predict the feature representation of an image from varied augmented views. Simultaneously, the multi-view encoder is improved using the exponential moving average of the single-view encoder. Contrastive learning is integrated into medical image fusion by creating a feature contrast space without constructing negative samples. This feature contrast space cleverly uses the information of the difference in the feature product of the source image and its corresponding augmented image. It continuously guides the network to constantly optimize its fusion effect by combining the method of structural similarity loss, to achieve more accurate and efficient image fusion. This approach represents an end-to-end unsupervised fusion model. Experimental validation shows that our proposed method demonstrates performance comparable to state-of-the-art techniques in both subjective evaluation and objective metrics.

Multimodal medical image fusion (MMIF) extracts the most meaningful information from multiple source images, enabling a more comprehensive and accurate diagnosis. Achieving high-quality fusion results requires a careful balance of brightness, color, contrast, and detail; this ensures that the fused images effectively display relevant anatomical structures and reflect the functional status of the tissues. However, existing MMIF methods have limited capacity to capture detailed features during conventional training and suffer from insufficient cross-modal feature interaction, leading to suboptimal fused image quality. To address these issues, this study proposes a two-stage diffusion model-based fusion network (DM-FNet) to achieve unified MMIF. In Stage I, a diffusion process trains UNet for image reconstruction. UNet captures detailed information through progressive denoising and represents multilevel data, providing a rich set of feature representations for the subsequent fusion network. In Stage II, noisy images at various steps are input into the fusion network to enhance the model's feature recognition capability. Three key fusion modules are also integrated to process medical images from different modalities adaptively. Ultimately, the robust network structure and a hybrid loss function are integrated to harmonize the fused image's brightness, color, contrast, and detail, enhancing its quality and information density. The experimental results across various medical image types demonstrate that the proposed method performs exceptionally well regarding objective evaluation metrics. The fused image preserves appropriate brightness, a comprehensive distribution of radioactive tracers, rich textures, and clear edges. The code is available athttps://github.com/HeDan-11/DM-FNet.

The combination of CT and MRI can provide more accurate images of lesions, yielding a significantly higher diagnostic value compared to single-modality pathological images. However, in CT-MRI fusion, preserving the gray-scale distribution of the source image while avoiding ‘detail halos’ poses a challenge. Therefore, we propose the utilization of anisotropic guided filtering (AnisGF), which exhibits excellent edge-preservation properties, to address structural inconsistencies in regions between the two modalities. The local neighborhood variance is utilized for optimizing the weight to achieve maximum diffusion, and subsequently decomposing the source image based on this criterion. A pre-trained convolutional neural network (CNN) is employed to accomplish the mapping from the source image to the weight map, while AnisGF is utilized for maintaining local consistency between them. The efficacy of this novel image fusion algorithm in preserving intricate details without compromising has been demonstrated through a combination of qualitative and quantitative experiments.

Sparse Coding (SC) and Convolution Sparse Coding (CSC) are two widely studied sparse methods in computer vision and signal processing. SC encodes the image patches independently, however fails to utilize the correlation among them. CSC adopts a convolution operator to connect the overlapping patches but in an inflexible manner. In this paper, a novel integrated framework for the two sparse models is proposed, wherein the local correlations among patches are controllable by manipulating a window function. Moreover, the inherent border effect of a convolution model is mitigated with a carefully designed weight function. It can be demonstrated that both SC and CSC are two distinct implementations of this framework. Consequently, our unified framework provides a balanced solution by addressing the strengths and limitations of both SC and CSC. Extensive experimental results are presented to demonstrate the superiority and effectiveness of the proposed method for image inpainting tasks.

Medical image fusion technology integrates the contents of medical images of different modalities, thereby assisting users of medical images to better understand their meaning. However, the fusion of medical images corrupted by noise remains a challenge. To solve the existing problems in medical image fusion and denoising algorithms related to excessive blur, unclean denoising, gradient information loss, and color distortion, a novel medical image fusion and denoising algorithm is proposed. First, a new image layer decomposition model based on hybrid variation-sparse representation and weighted Schatten p-norm is proposed. The alternating direction method of multipliers is used to update the structure, detail layer dictionary, and detail layer coefficient map of the input image while denoising. Subsequently, appropriate fusion rules are employed for the structure layers and detail layer coefficient maps. Finally, the fused image is restored using the fused structure layer, detail layer dictionary, and detail layer coefficient maps. A large number of experiments confirm the superiority of the proposed algorithm over other algorithms. The proposed medical image fusion and denoising algorithm can effectively remove noise while retaining the gradient information without color distortion.