• A new method, UNISELF, is proposed to improve multiple sclerosis lesion segmentation. • UNISELF uses self-ensembled lesion fusion to improve accuracy and generalization. • UNISELF uses test-time instance normalization to address latent feature distribution shift. • UNISELF is among the top methods in the ISBI lesion segmentation challenge. • UNISELF outperforms other benchmarks on various out-of-domain test datasets. Automated segmentation of multiple sclerosis (MS) lesions using multicontrast magnetic resonance (MR) images improves efficiency and reproducibility compared to manual delineation, with deep learning (DL) methods achieving state-of-the-art performance. However, these DL-based methods have yet to simultaneously optimize in-domain accuracy and out-of-domain generalization when trained on a single source with limited data, or their performance has been unsatisfactory. To fill this gap, we propose a method called UNISELF, which achieves high accuracy within a single training domain while demonstrating strong generalizability across multiple out-of-domain test datasets. UNISELF employs a novel test-time self-ensembled lesion fusion to improve segmentation accuracy, and leverages test-time instance normalization (TTIN) of latent features to address domain shifts and missing input contrasts. Trained on the ISBI 2015 longitudinal MS segmentation challenge training dataset, UNISELF ranks among the best-performing methods on the challenge test dataset. Additionally, UNISELF outperforms all benchmark methods trained on the same ISBI training data across diverse out-of-domain test datasets with domain shifts and missing contrasts, including the public MICCAI 2016 and UMCL datasets, as well as a private multisite dataset. These test datasets exhibit domain shifts and/or missing contrasts caused by variations in acquisition protocols, scanner types, and imaging artifacts arising from imperfect acquisition. Our code is available at https://github.com/Jinwei1209/UNISELF .

Tagged magnetic resonance imaging (tMRI) is a valuable tool for visualizing and quantifying tissue deformation in vivo. Its use is often hampered, however, by tag fading, long computation times, and the challenge of ensuring diffeomorphic, incompressible motion fields. In this paper, we describe a novel integration of the harmonic phase (HARP) approach to tMRI analysis with an unsupervised deep learning-based registration framework to estimate 2D and 3D motion fields that are diffeomorphic and nearly incompressible. The resulting method, called deep sinusoidally transformed HARP, or DSHARP, enables end-to-end network training by implementing a transformation of the harmonic phase to remove phase-wrapping discontinuities. It produces diffeomorphic motion by estimating a stationary velocity field from which motion is computed using the scaling and squaring technique. Finally, it encourages incompressibility using a novel Jacobian determinant loss term during network training. We evaluated DSHARP on 2D and 3D phantom data with simulated incompressible motions, real 3D human tongue data acquired during speech from both healthy and glossectomy subjects, and cardiac tagged MRI from the public STACOM 2011 benchmark. Our approach outperforms HARP, SinMod, SyN, PVIRA, VoxelMorph, and DeepTag in tracking accuracy, computation speed, and preservation of incompressibility.

Medical image harmonization aims to reduce the differences in appearance caused by scanner hardware variations to allow for consistent and reliable comparisons across devices. Harmonization based on paired images from different devices has limited applicability in real-world clinical settings. On the other hand, unpaired harmonization typically does not guarantee anatomy consistency, which is problematic because anatomical information preservation is paramount. The Schrödinger bridge framework has achieved state-of-the-art style transfer performance with natural images by matching distributions of unpaired images, but this approach can also introduce anatomy changes when applied to medical images. We show that such changes occur because the Schrödinger bridge uses the square of the Euclidean distance between images as the transport cost in an entropy-regularized optimal transport problem. Such a transport cost is not appropriate for measuring anatomical distances, as medical images with the same anatomy need not have a small Euclidean distance between them. In this paper, we propose a latent metric Schrödinger bridge (LMSB) framework to improve the anatomical consistency for the harmonization of medical images. We develop an invertible network that maps medical images into a latent Euclidean metric space where the distances among images with the same anatomy are minimized using the pullback latent metric. Within this latent space, we train a Schrödinger bridge to match distributions. We show that the proposed LMSB is superior to the direct application of a Schrödinger bridge to harmonize optical coherence tomography (OCT) images.

Abstract Producing spatial transformations that are diffeomorphic is a key goal in deformable image registration. As a diffeomorphic transformation should have positive Jacobian determinant $$\vert J\vert $$ | J | everywhere, the number of pixels (2D) or voxels (3D) with $$\vert J\vert <0$$ | J | < 0 has been used to test for diffeomorphism and also to measure the irregularity of the transformation. For digital transformations, $$\vert J\vert $$ | J | is commonly approximated using a central difference, but this strategy can yield positive $$\vert J\vert $$ | J | ’s for transformations that are clearly not diffeomorphic—even at the pixel or voxel resolution level. To show this, we first investigate the geometric meaning of different finite difference approximations of $$\vert J\vert $$ | J | . We show that to determine if a deformation is diffeomorphic for digital images, the use of any individual finite difference approximation of $$\vert J\vert $$ | J | is insufficient. We further demonstrate that for a 2D transformation, four unique finite difference approximations of $$\vert J\vert $$ | J | ’s must be positive to ensure that the entire domain is invertible and free of folding at the pixel level. For a 3D transformation, ten unique finite differences approximations of $$\vert J\vert $$ | J | ’s are required to be positive. Our proposed digital diffeomorphism criteria solves several errors inherent in the central difference approximation of $$\vert J\vert $$ | J | and accurately detects non-diffeomorphic digital transformations. The source code of this work is available at https://github.com/yihao6/digital_diffeomorphism .

Optical coherence tomography angiography (OCTA) is an imaging modality that can be used for analyzing retinal vasculature. Quantitative assessment of en face OCTA images requires accurate segmentation of the capillaries. Using deep learning approaches for this task faces two major challenges. First, acquiring sufficient manual delineations for training can take hundreds of hours. Second, OCTA images suffer from numerous contrast-related artifacts that are currently inherent to the modality and vary dramatically across scanners. We propose to solve both problems by learning a disentanglement of an anatomy component and a local contrast component from paired OCTA scans. With the contrast removed from the anatomy component, a deep learning model that takes the anatomy component as input can learn to segment vessels with a limited portion of the training images being manually labeled. Our method demonstrates state-of-the-art performance for OCTA vessel segmentation.

Synthesizing a CT image from an available MR image has recently emerged as a key goal in radiotherapy treatment planning for cancer patients. CycleGANs have achieved promising results on unsupervised MR-to-CT image synthesis; however, because they have no direct constraints between input and synthetic images, cycleGANs do not guarantee structural consistency between these two images. This means that anatomical geometry can be shifted in the synthetic CT images, clearly a highly undesirable outcome in the given application. In this paper, we propose a structure-constrained cycleGAN for unsupervised MR-to-CT synthesis by defining an extra structure-consistency loss based on the modality independent neighborhood descriptor. We also utilize a spectral normalization technique to stabilize the training process and a self-attention module to model the long-range spatial dependencies in the synthetic images. Results on unpaired brain and abdomen MR-to-CT image synthesis show that our method produces better synthetic CT images in both accuracy and visual quality as compared to other unsupervised synthesis methods. We also show that an approximate affine pre-registration for unpaired training data can improve synthesis results.

It is faster and therefore cheaper to acquire magnetic resonance images (MRI) with higher in-plane resolution than through-plane resolution. The low resolution of such acquisitions can be increased using post-processing techniques referred to as super-resolution (SR) algorithms. SR is known to be an ill-posed problem. Most state-of-the-art SR algorithms rely on the presence of external/training data to learn a transform that converts low resolution input to a higher resolution output. In this paper an SR approach is presented that is not dependent on any external training data and is only reliant on the acquired image. Patches extracted from the acquired image are used to estimate a set of new images, where each image has increased resolution along a particular direction. The final SR image is estimated by combining images in this set via the technique of Fourier Burst Accumulation. Our approach was validated on simulated low resolution MRI images, and showed significant improvement in image quality and segmentation accuracy when compared to competing SR methods. SR of FLuid Attenuated Inversion Recovery (FLAIR) images with lesions is also demonstrated.

Quantitative measurements from segmentations of human brain magnetic resonance (MR) images provide important biomarkers for normal aging and disease progression. In this paper, we propose a patch-based tissue classification method from MR images that uses a sparse dictionary learning approach and atlas priors. Training data for the method consists of an atlas MR image, prior information maps depicting where different tissues are expected to be located, and a hard segmentation. Unlike most atlas-based classification methods that require deformable registration of the atlas priors to the subject, only affine registration is required between the subject and training atlas. A subject-specific patch dictionary is created by learning relevant patches from the atlas. Then the subject patches are modeled as sparse combinations of learned atlas patches leading to tissue memberships at each voxel. The combination of prior information in an example-based framework enables us to distinguish tissues having similar intensities but different spatial locations. We demonstrate the efficacy of the approach on the application of whole-brain tissue segmentation in subjects with healthy anatomy and normal pressure hydrocephalus, as well as lesion segmentation in multiple sclerosis patients. For each application, quantitative comparisons are made against publicly available state-of-the art approaches.

The performance of image analysis algorithms applied to magnetic resonance images is strongly influenced by the pulse sequences used to acquire the images. Algorithms are typically optimized for a targeted tissue contrast obtained from a particular implementation of a pulse sequence on a specific scanner. There are many practical situations, including multi-institution trials, rapid emergency scans, and scientific use of historical data, where the images are not acquired according to an optimal protocol or the desired tissue contrast is entirely missing. This paper introduces an image restoration technique that recovers images with both the desired tissue contrast and a normalized intensity profile. This is done using patches in the acquired images and an atlas containing patches of the acquired and desired tissue contrasts. The method is an example-based approach relying on sparse reconstruction from image patches. Its performance in demonstrated using several examples, including image intensity normalization, missing tissue contrast recovery, automatic segmentation, and multimodal registration. These examples demonstrate potential practical uses and also illustrate limitations of our approach.