In volume-to-volume translations in medical images, existing models often struggle to capture the inherent volumetric distribution using 3D voxel-space representations, due to high computational dataset demands. We present Score-Fusion, a novel volumetric translation model that effectively learns 3D representations by ensembling perpendicularly trained 2D diffusion models in score function space. By carefully initializing our model to start with an average of 2D models as in existing models, we reduce 3D training to a fine-tuning process, mitigating computational and data demands. Furthermore, we explicitly design the 3D model’s hierarchical layers to learn ensembles of 2D features, further enhancing efficiency and performance. Moreover, Score-Fusion naturally extends to multi-modality settings by fusing diffusion models conditioned on different inputs for flexible, accurate integration. We demonstrate that 3D representation is essential for better performance in downstream recognition tasks, such as tumor segmentation, where most segmentation models are based on 3D representation. Extensive experiments demonstrate that Score-Fusion achieves superior accuracy and volumetric fidelity in 3D medical image super-resolution and modality translation. Additionally, we extend Score-Fusion to video super-resolution by integrating 2D diffusion models on time-space slices with a spatial-temporal video diffusion backbone, highlighting its potential for general-purpose volume translation and providing broader insight into learning-based approaches for score function fusion.

Multimodal image translation has found useful applications in solving several medical imaging problems. In this paper, we presented a systematic analysis of multimodal images and machine learning-based image translation from an information-theoretic perspective. Specifically, we analyzed the amount of mutual information that exists in some commonly used multimodal images. This analysis revealed varying structural correlation across modalities and tissue-dependence of mutual information. We also analyzed the amount of information transferred and gained in multimodal image translation and provided an upper bound on the information gain. Information-theoretic measures were also proposed to assess the effectiveness of an image translator, and the uncertainty associated with image translation. Numerical results were presented to demonstrate the information gain in practical multimodal image translation, and to validate the proposed upper bound on information gain and the translation error predictor. Finally, several potential applications of our analysis results were discussed, including the image denoising and reconstruction using side information generated by image translation. The findings from this study may prove useful for guiding the further development and application of multimodal image translation.

Image reconstruction from limited and/or sparse data is known to be an ill-posed problem and a priori information/constraints have played an important role in solving the problem. Early constrained image reconstruction methods utilize image priors based on general image properties such as sparsity, low-rank structures, spatial support bound, etc. Recent deep learning-based reconstruction methods promise to produce even higher quality reconstructions by utilizing more specific image priors learned from training data. However, learning high-dimensional image priors requires huge amounts of training data that are currently not available in medical imaging applications. As a result, deep learning-based reconstructions often suffer from two known practical issues: a) sensitivity to data perturbations (e.g., changes in data sampling scheme), and b) limited generalization capability (e.g., biased reconstruction of lesions). This paper proposes a new method to address these issues. The proposed method synergistically integrates model-based and data-driven learning in three key components. The first component uses the linear vector space framework to capture global dependence of image features; the second exploits a deep network to learn the mapping from a linear vector space to a nonlinear manifold; the third is an unrolling-based deep network that captures local residual features with the aid of a sparsity model. The proposed method has been evaluated with magnetic resonance imaging data, demonstrating improved reconstruction in the presence of data perturbation and/or novel image features. The method may enhance the practical utility of deep learning-based image reconstruction.

Deuterium magnetic resonance spectroscopic imaging (DMRSI) has recently been recognized as a potentially powerful tool for noninvasive imaging of brain energy metabolism and tumor. However, the low sensitivity of DMRSI has significantly limited its utility for both research and clinical applications. This work presents a novel machine learning-based method to address this limitation. The proposed method synergistically integrates physics-based subspace modeling and data-driven deep learning for effective denoising, making high-resolution dynamic DMRSI possible. Specifically, a novel subspace model was used to represent the dynamic DMRSI signals; deep neural networks were trained to capture the low-dimensional manifolds of the spectral and temporal distributions of practical dynamic DMRSI data. The learned subspace and manifold structures were integrated via a regularization formulation to remove measurement noise. Theoretical analysis, computer simulations, and in vivo experiments have been conducted to demonstrate the denoising efficacy of the proposed method which enabled high-resolution imaging capability. The translational potential was demonstrated in tumor-bearing rats, where the Warburg effect associated with cancer metabolism and tumor heterogeneity were successfully captured. The new method may not only provide an effective tool to enhance the sensitivity of DMRSI for basic research and clinical applications but also provide a framework for denoising other spatiospectral data.