Fully automated myocardial segmentation from cardiac magnetic resonance imaging (MRI) is vital for efficient diagnosis and treatment planning. Although numerous automated methods have been proposed, they typically focus on single MRI sequences and therefore have difficulties in generalizing across vendors and across cardiac MRI protocols. Simultaneous analysis of complementary cardiac MRI sequences, such as cine, T1 mapping, and late gadolinium enhancement (LGE) MRI, remains challenging due to their distinct image characteristics and scanner-specific variations. To address these issues, we propose an unsupervised domain adaptation approach that allows robust myocardial segmentation across multi-vendor cine, T1, and LGE MRI data. In particular, we introduce a class-imbalance self-training framework to transfer information learned from a source domain with labels to any unlabeled target domain, while maintaining consistent performance across different MRI sequences. Our framework iteratively refines segmentation accuracy by generating pseudo-labels for target data using a hardness-aware strategy, thus effectively addressing the problem of class imbalance in cardiac MRI segmentation. To mitigate data scarcity following pseudo-label selection, we employ a variance-guided vicinal feature extrapolation, which expands data points in the feature space into a probabilistic distribution. This, in turn, facilitates joint source-target training by generating a larger intersection in the feature space. Experimental results demonstrate that our framework outperforms existing methods when assessed using the Dice coefficient and Hausdorff distance. Our framework enables cardiac evaluation across MRI protocols without sequence-specific manual annotations.

Tagged magnetic resonance imaging (tMRI) is a valuable tool for visualizing and quantifying tissue deformation in vivo. Its use is often hampered, however, by tag fading, long computation times, and the challenge of ensuring diffeomorphic, incompressible motion fields. In this paper, we describe a novel integration of the harmonic phase (HARP) approach to tMRI analysis with an unsupervised deep learning-based registration framework to estimate 2D and 3D motion fields that are diffeomorphic and nearly incompressible. The resulting method, called deep sinusoidally transformed HARP, or DSHARP, enables end-to-end network training by implementing a transformation of the harmonic phase to remove phase-wrapping discontinuities. It produces diffeomorphic motion by estimating a stationary velocity field from which motion is computed using the scaling and squaring technique. Finally, it encourages incompressibility using a novel Jacobian determinant loss term during network training. We evaluated DSHARP on 2D and 3D phantom data with simulated incompressible motions, real 3D human tongue data acquired during speech from both healthy and glossectomy subjects, and cardiac tagged MRI from the public STACOM 2011 benchmark. Our approach outperforms HARP, SinMod, SyN, PVIRA, VoxelMorph, and DeepTag in tracking accuracy, computation speed, and preservation of incompressibility.

Accurate survival prediction using multimodal magnetic resonance imaging (MRI) plays a crucial role in clinical decision-making for patients with glioblastoma (GBM). In this work, we propose a multimodal framework, GlioSurvNet, that integrates deep learning features extracted from Swin UNETR and clinical variables to predict patient survival. Our framework makes use of multiple MRI sequences, including T1, T1 with contrast enhancement, T2-weighted, and FLAIR MRI, to capture diverse tumor characteristics. The Swin UNETR architecture simultaneously carries out tumor segmentation and extracts hierarchical features from multimodal MRI data. These deep learning features are then combined with clinical variables, which are input into a multi-layer perceptron network to yield survival probabilities. We evaluated our framework on a cohort of 287 patients from two independent databases, UPENN-GBM and UCSF-PDGM, demonstrating superior survival prediction performance when compared with existing methods. Our framework achieved a time-dependent concordance index of 0.693 and an integrated brier score of 0.14 with improved risk stratification. GlioSurvNet offers a robust tool for personalized prognosis and treatment planning in GBM patients.

Conventional unsupervised domain adaptation (UDA) requires access to source data and/or source model parameters, prohibiting its practical application in terms of privacy, security, and intellectual property. Recent black-box UDA (BDA) reduces such constraints by defining a pseudo label from a single encapsulated source application programming interface (API) prediction, which allows for self-training of the target model. Nonetheless, existing methods have limited consideration for multi-source settings, in which multiple source domain APIs are available to generate pseudo labels. In this work, we introduce a novel training framework for multi-source BDA (MSBDA), dubbed Label Space-Induced Pseudo Label Refinement (LPR). Specifically, LPR incorporates a Pseudo label Refinery Network (PRN) that learns the relationship among source domains conditioned by the target domain only utilizing source API's prediction. The target model is adapted by our dual phases PRN. First, a warm-up phase targets to avoid failure due to noisy samples and provide an initial pseudo-label, which is followed by a label refinement phase with domain relationship exploration. We provide theoretical support for the mechanism of the LPR. Experimental results on four benchmark datasets demonstrate that MSBDA using LPR achieves competitive performance compared to state-of-the-art approaches with different DA settings.

Magnetic Resonance Imaging (MRI) allows analyzing speech production by capturing high-resolution images of the dynamic processes in the vocal tract. In clinical applications, combining MRI with synchronized speech recordings leads to improved patient outcomes, especially if a phonological-based approach is used for assessment. However, when audio signals are unavailable, the recognition accuracy of sounds is decreased when using only MRI data. We propose a contrastive learning approach to improve the detection of phonological classes from MRI data when acoustic signals are not available at inference time. We demonstrate that frame-wise recognition of phonological classes improves from an f1 of 0.74 to 0.85 when the contrastive loss approach is implemented. Furthermore, we show the utility of our approach in the clinical application of using such phonological classes to assess speech disorders in patients with tongue cancer, yielding promising results in the recognition task.

Unsupervised domain adaptation (UDA) has been successfully applied to transfer knowledge from a labeled source domain to target domains without their labels. Recently introduced transferable prototypical networks (TPNs) further address class-wise conditional alignment. In TPN, while the closeness of class centers between source and target domains is explicitly enforced in a latent space, the underlying fine-grained subtype structure and the cross-domain within-class compactness have not been fully investigated. To counter this, we propose a new approach to adaptively perform a fine-grained subtype-aware alignment to improve the performance in the target domain without the subtype label in both domains. The insight of our approach is that the unlabeled subtypes in a class have the local proximity within a subtype while exhibiting disparate characteristics because of different conditional and label shifts. Specifically, we propose to simultaneously enforce subtype-wise compactness and class-wise separation, by utilizing intermediate pseudo-labels. In addition, we systematically investigate various scenarios with and without prior knowledge of subtype numbers and propose to exploit the underlying subtype structure. Furthermore, a dynamic queue framework is developed to evolve the subtype cluster centroids steadily using an alternative processing scheme. Experimental results, carried out with multiview congenital heart disease data and VisDA and DomainNet, show the effectiveness and validity of our subtype-aware UDA, compared with state-of-the-art UDA methods.

Finite element models (FEM) of the tongue have facilitated speech studies through analysis of internal muscle forces indirectly derived from imaging data. In this work, we build a uniform hexahedral FEM of a tongue atlas constructed from magnetic resonance imaging data of a healthy population. The FEM is driven by inverse internal tongue tissue kinematics of speakers temporally aligned and deformed into the same atlas space, while performing the speech task "a souk" allowing muscle activation predictions. This work aims to investigate the commonalities in tongue motor strategies in the articulation of "a souk" predicted by the inverse tongue atlas model. Our findings report variability among five speakers for estimated muscle activations with a similarity index using a dynamic time warp function. Two speakers show similarity index > 0.9 and two others < 0.7 with respect to a reference speaker for most tongue muscles. The relative motion tracking error of the model is less than 2% which is promising for speech study applications.

Multimodal representation learning using visual movements from cine magnetic resonance imaging (MRI) and their acoustics has shown great potential to learn shared representation and to predict one modality from another. Here, we propose a new synthesis framework to translate from cine MRI sequences to spectrograms with a limited dataset size. Our framework hinges on a novel fully convolutional heterogeneous translator, with a 3D CNN encoder for efficient sequence encoding and a 2D transpose convolution decoder. In addition, a pairwise correlation of the samples with the same speech word is utilized with a latent space representation disentanglement scheme. Furthermore, an adversarial training approach with generative adversarial networks is incorporated to provide enhanced realism on our generated spectrograms. Our experimental results, carried out with a total of 63 cine MRI sequences alongside speech acoustics, show that our framework improves synthesis accuracy, compared with competing methods. Our framework thereby has shown the potential to aid in better understanding the relationship between the two modalities.

Deep learning has great potential for accurate detection and classification of diseases with medical imaging data, but the performance is often limited by the number of training datasets and memory requirements. In addition, many deep learning models are considered a "black-box," thereby often limiting their adoption in clinical applications. To address this, we present a successive subspace learning model, termed VoxelHop, for accurate classification of Amyotrophic Lateral Sclerosis (ALS) using T2-weighted structural MRI data. Compared with popular convolutional neural network (CNN) architectures, VoxelHop has modular and transparent structures with fewer parameters without any backpropagation, so it is well-suited to small dataset size and 3D imaging data. Our VoxelHop has four key components, including (1) sequential expansion of near-to-far neighborhood for multi-channel 3D data; (2) subspace approximation for unsupervised dimension reduction; (3) label-assisted regression for supervised dimension reduction; and (4) concatenation of features and classification between controls and patients. Our experimental results demonstrate that our framework using a total of 20 controls and 26 patients achieves an accuracy of 93.48 % and an AUC score of 0.9394 in differentiating patients from controls, even with a relatively small number of datasets, showing its robustness and effectiveness. Our thorough evaluations also show its validity and superiority to the state-of-the-art 3D CNN classification approaches. Our framework can easily be generalized to other classification tasks using different imaging modalities.

Modeling statistical properties of anatomical structures using magnetic resonance imaging is essential for revealing common information of a target population and unique properties of specific subjects. In brain imaging, a statistical brain atlas is often constructed using a number of healthy subjects. When tumors are present, however, it is difficult to either provide a common space for various subjects or align their imaging data due to the unpredictable distribution of lesions. Here we propose a deep learning-based image inpainting method to replace the tumor regions with normal tissue intensities using only a patient population. Our framework has three major innovations: 1) incompletely distributed datasets with random tumor locations can be used for training; 2) irregularly-shaped tumor regions are properly learned, identified, and corrected; and 3) a symmetry constraint between the two brain hemispheres is applied to regularize inpainted regions. Henceforth, regular atlas construction and image registration methods can be applied using inpainted data to obtain tissue deformation, thereby achieving group-specific statistical atlases and patient-to-atlas registration. Our framework was tested using the public database from the Multimodal Brain Tumor Segmentation challenge. Results showed increased similarity scores as well as reduced reconstruction errors compared with three existing image inpainting methods. Patient-to-atlas registration also yielded better results with improved normalized cross-correlation and mutual information and a reduced amount of deformation over the tumor regions.

Recent advances in unsupervised domain adaptation (UDA) show that transferable prototypical learning presents a powerful means for class conditional alignment, which encourages the closeness of cross-domain class centroids. However, the cross-domain inner-class compactness and the underlying fine-grained subtype structure remained largely underexplored. In this work, we propose to adaptively carry out the fine-grained subtype-aware alignment by explicitly enforcing the class-wise separation and subtype-wise compactness with intermediate pseudo labels. Our key insight is that the unlabeled subtypes of a class can be divergent to one another with different conditional and label shifts, while inheriting the local proximity within a subtype. The cases with or without the prior information on subtype numbers are investigated to discover the underlying subtype structure in an online fashion. The proposed subtype-aware dynamic UDA achieves promising results on a medical diagnosis task.

In this work, we propose an adversarial unsupervised domain adaptation (UDA) method under inherent conditional and label shifts, in which we aim to align the distributions w.r.t. both p(x|y) and p(y). Since labels are inaccessible in a target domain, conventional adversarial UDA methods assume that p(y) is invariant across domains and rely on aligning p(x) as an alternative to the p(x|y) alignment. To address this, we provide a thorough theoretical and empirical analysis of the conventional adversarial UDA methods under both conditional and label shifts, and propose a novel and practical alternative optimization scheme for adversarial UDA. Specifically, we infer the marginal p(y) and align p(x|y) iteratively at the training stage, and precisely align the posterior p(y|x) at the testing stage. Our experimental results demonstrate its effectiveness on both classification and segmentation UDA and partial UDA.

In this work, we propose a domain generalization (DG) approach to learn on several labeled source domains and transfer knowledge to a target domain that is inaccessible in training. Considering the inherent conditional and label shifts, we would expect the alignment of p(x|y) and p(y). However, the widely used domain invariant feature learning (IFL) methods relies on aligning the marginal concept shift w.r.t. p(x), which rests on an unrealistic assumption that p(y) is invariant across domains. We thereby propose a novel variational Bayesian inference framework to enforce the conditional distribution alignment w.r.t. p(x|y) via the prior distribution matching in a latent space, which also takes the marginal label shift w.r.t. p(y) into consideration with the posterior alignment. Extensive experiments on various benchmarks demonstrate that our framework is robust to the label shift and the cross-domain accuracy is significantly improved, thereby achieving superior performance over the conventional IFL counterparts.

Semantic segmentation is a class of methods to classify each pixel in an image into semantic classes, which is critical for autonomous vehicles and surgery systems. Cross-entropy (CE) loss-based deep neural networks (DNN) achieved great success w.r.t. the accuracy-based metrics, e.g., mean Intersection-over Union. However, the CE loss has a limitation in that it ignores varying degrees of severity of pair-wise misclassified results. For instance, classifying a car into the road is much more terrible than recognizing it as a bus. To sidestep this, in this work, we propose to incorporate the severity-aware inter-class correlation into our Wasserstein training framework by configuring its ground distance matrix. In addition, our method can adaptively learn the ground metric in a high-fidelity simulator, following a reinforcement alternative optimization scheme. We evaluate our method using the CARLA simulator with the Deeplab backbone, demonstraing that our method significantly improves the survival time in the CARLA simulator. In addition, our method can be readily applied to existing DNN architectures and algorithms while yielding superior performance. We report results from experiments carried out with the CamVid and Cityscapes datasets.

Muscle coordination patterns of lingual behaviors are synergies generated by deforming local muscle groups in a variety of ways. Functional units are functional muscle groups of local structural elements within the tongue that compress, expand, and move in a cohesive and consistent manner. Identifying the functional units using tagged-magnetic resonance imaging (MRI) sheds light on the mechanisms of normal and pathological muscle coordination patterns, yielding improvement in surgical planning, treatment, or rehabilitation procedures. In this paper, to mine this information, we propose a matrix factorization and probabilistic graphical model framework to produce building blocks and their associated weighting map using motion quantities extracted from tagged-MRI. Our tagged-MRI imaging and accurate voxel-level tracking provide previously unavailable internal tongue motion patterns, thus revealing the inner workings of the tongue during speech or other lingual behaviors. We then employ spectral clustering on the weighting map to identify the cohesive regions defined by the tongue motion that may involve multiple or undocumented regions. To evaluate our method, we perform a series of experiments. We first use two-dimensional images and synthetic data to demonstrate the accuracy of our method. We then use three-dimensional synthetic and in vivo tongue motion data using protrusion and simple speech tasks to identify subject-specific and data-driven functional units of the tongue in localized regions.

Tagged magnetic resonance imaging has been used for decades to observe and quantify motion and strain of deforming tissue. It is challenging to obtain 3-D motion estimates due to a tradeoff between image slice density and acquisition time. Typically, interpolation methods are used either to combine 2-D motion extracted from sparse slice acquisitions into 3-D motion or to construct a dense volume from sparse acquisitions before image registration methods are applied. This paper proposes a new phase-based 3-D motion estimation technique that first computes harmonic phase volumes from interpolated tagged slices and then matches them using an image registration framework. The approach uses several concepts from diffeomorphic image registration with a key novelty that defines a symmetric similarity metric on harmonic phase volumes from multiple orientations. The material property of harmonic phase solves the aperture problem of optical flow and intensity-based methods and is robust to tag fading. A harmonic magnitude volume is used in enforcing incompressibility in the tissue regions. The estimated motion fields are dense, incompressible, diffeomorphic, and inverse-consistent at a 3-D voxel level. The method was evaluated using simulated phantoms, human brain data in mild head accelerations, human tongue data during speech, and an open cardiac data set. The method shows comparable accuracy to three existing methods while demonstrating low computation time and robustness to tag fading and noise.

Multimodal image registration is a class of algorithms to find correspondence from different modalities. Since different modalities do not exhibit the same characteristics, finding accurate correspondence still remains a challenge. To deal with this, mutual information (MI)-based registration has been a preferred choice as MI is based on the statistical relationship between both volumes to be registered. However, MI has some limitations. First, MI-based registration often fails when there are local intensity variations in the volumes. Second, MI only considers the statistical intensity relationships between both volumes and ignores the spatial and geometric information about the voxel. In this work, we propose to address these limitations by incorporating spatial and geometric information via a 3D Harris operator. In particular, we focus on the registration between a high-resolution image and a low-resolution image. The MI cost function is computed in the regions where there are large spatial variations such as corner or edge. In addition, the MI cost function is augmented with geometric information derived from the 3D Harris operator applied to the high-resolution image. The robustness and accuracy of the proposed method were demonstrated using experiments on synthetic and clinical data including the brain and the tongue. The proposed method provided accurate registration and yielded better performance over standard registration methods.

Glossectomy changes properties of the tongue and negatively affects patients' speech production. Among the most difficult consonants to produce in the post-glossectomy speakers, the sibilant fricatives /s/ and /sh/ are often problematic. To better understand these problems in production, this study analyzed acoustic and articulatory data of /s/ and /sh/ from three subjects: one normal speaker and two post-glossectomy speakers with abnormal /s/ or /sh. Based on cine magnetic resonance images, three dimensional vocal tract reconstructions, tongue surface shapes behind constrictions, and area functions were analyzed. Our results show that in each patient, contrary to normal, /s/ and /sh/ were quite similar in acoustic spectra, tongue surface shapes, and constriction locations. In the abnormal /s/, the missing unilateral tongue tissue created an air flow bypass which made the constriction further backward. The abnormal /sh/ may be explained by the lack of precise tongue control after surgery. In addition, the tongue surfaces in the patients were more asymmetric in the back and were not grooved for /s/ anterior to the constriction.