Current cardiac cine magnetic resonance image (cMR) studies focus on the end diastole (ED) and end systole (ES) phases, while ignoring the abundant temporal information in the whole image sequence. This is because whole sequence segmentation is currently a tedious process and in-accurate. Conventional whole sequence segmentation approaches first estimate the motion field between frames, which is then used to propagate the mask along the temporal axis. However, the mask propagation results could be prone to error, especially for the basal and apex slices, where through-plane motion leads to significant morphology and structural change during the cardiac cycle. Inspired by recent advances in video object segmentation (VOS), based on spatiotemporal memory (STM) networks, we propose a continuous STM (CSTM) network for semi-supervised whole heart and whole sequence cMR segmentation. Our CSTM network takes full advantage of the spatial, scale, temporal and through-plane continuity prior of the underlying heart anatomy structures, to achieve accurate and fast 4D segmentation. Results of extensive experiments across multiple cMR datasets show that our method can improve the 4D cMR segmentation performance, especially for the hard-to-segment regions. Project page is at https://github.com/DeepTag/CSTM.

Multi-modal registration between tagged and untagged cardiac cine magnetic resonance (MR) images remains difficult, due to the domain gap and large deformations between the two modalities. Recent work using an image-to-image translation (I2I) module to overcome the domain gap can convert the multi-modal into a mono-modal registration task and take advantage of advanced mono-modal registration architectures. However, they often ignore two issues: the sample-specific style of each image to be registered during I2I and large hybrid rigid and non-rigid deformations between modalities. We first propose an exemplar-based I2I module capable of unsupervised cross-domain correspondence learning to enforce the style consistency between the fake image and the image to be registered. Then we propose an efficient cascaded vision transformer-based registration network to predict both the affine and non-rigid deformations, in which a single feature embedding subnetwork is shared by the two stages of deformation prediction. We validated our method on a clinical cardiac MR dataset with paired but unaligned untagged and tagged MR images. The results show that our method outperforms traditional methods significantly in terms of the I2I quality and multi-modal image registration accuracy.

We propose a novel neural deformable model (NDM) targeting at the reconstruction and modeling of 3D bi-ventricular shape of the heart from 2D sparse cardiac magnetic resonance (CMR) imaging data. We model the bi-ventricular shape using blended deformable superquadrics, which are parameterized by a set of geometric parameter functions and are capable of deforming globally and locally. While global geometric parameter functions and deformations capture gross shape features from visual data, local deformations, parameterized as neural diffeomorphic point flows, can be learned to recover the detailed heart shape. Different from iterative optimization methods used in conventional deformable model formulations, NDMs can be trained to learn such geometric parameter functions, global and local deformations from a shape distribution manifold. Our NDM can learn to densify a sparse cardiac point cloud with arbitrary scales and generate high-quality triangular meshes automatically. It also enables the implicit learning of dense correspondences among different heart shape instances for accurate cardiac shape registration. Furthermore, the parameters of NDM are intuitive, and can be used by a physician without sophisticated post-processing. Experimental results on a large CMR dataset demonstrate the improved performance of NDM over conventional methods.

Modern medical imaging techniques, such as ultrasound (US) and cardiac magnetic resonance (MR) imaging, have enabled the evaluation of myocardial deformation directly from an image sequence. While many traditional cardiac motion tracking methods have been developed for the automated estimation of the myocardial wall deformation, they are not widely used in clinical diagnosis, due to their lack of accuracy and efficiency. In this paper, we propose a novel deep learning-based fully unsupervised method, SequenceMorph, for in vivo motion tracking in cardiac image sequences. In our method, we introduce the concept of motion decomposition and recomposition. We first estimate the inter-frame (INF) motion field between any two consecutive frames, by a bi-directional generative diffeomorphic registration neural network. Using this result, we then estimate the Lagrangian motion field between the reference frame and any other frame, through a differentiable composition layer. Our framework can be extended to incorporate another registration network, to further reduce the accumulated errors introduced in the INF motion tracking step, and to refine the Lagrangian motion estimation. By utilizing temporal information to perform reasonable estimations of spatio-temporal motion fields, this novel method provides a useful solution for image sequence motion tracking. Our method has been applied to US (echocardiographic) and cardiac MR (untagged and tagged cine) image sequences; the results show that SequenceMorph is significantly superior to conventional motion tracking methods, in terms of the cardiac motion tracking accuracy and inference efficiency.

Cardiac tagging magnetic resonance imaging (t-MRI) is the gold standard for regional myocardium deformation and cardiac strain estimation. However, this technique has not been widely used in clinical diagnosis, as a result of the difficulty of motion tracking encountered with t-MRI images. In this paper, we propose a novel deep learning-based fully unsupervised method for in vivo motion tracking on t-MRI images. We first estimate the motion field (INF) between any two consecutive t-MRI frames by a bi-directional generative diffeomorphic registration neural network. Using this result, we then estimate the Lagrangian motion field between the reference frame and any other frame through a differentiable composition layer. By utilizing temporal information to perform reasonable estimations on spatiotemporal motion fields, this novel method provides a useful solution for motion tracking and image registration in dynamic medical imaging. Our method has been validated on a representative clinical t-MRI dataset; the experimental results show that our method is superior to conventional motion tracking methods in terms of landmark tracking accuracy and inference efficiency. Project page is at: https://github.com/DeepTag/cardiac_tagging_motion_estimation.

Myocardial strain is a critical indicator of many cardiac diseases and dysfunctions. The goal of this paper is to extract and use the myocardial strain pattern from tagged magnetic resonance imaging (MRI) to identify and localize regional abnormal cardiac function in human subjects. In order to extract the myocardial strains from the tagged images, we developed a novel nontracking-based strain estimation method for tagged MRI. This method is based on the direct extraction of tag deformation, and therefore avoids some limitations of conventional displacement or tracking-based strain estimators. Based on the extracted spatio-temporal strain patterns, we have also developed a novel tensor-based classification framework that better conserves the spatio-temporal structure of the myocardial strain pattern than conventional vector-based classification algorithms. In addition, the tensor-based projection function keeps more of the information of the original feature space, so that abnormal tensors in the subspace can be back-projected to reveal the regional cardiac abnormality in a more physically meaningful way. We have tested our novel methods on 41 human image sequences, and achieved a classification rate of 87.80%. The regional abnormalities recovered from our algorithm agree well with the patient's pathology and clinical image interpretation, and provide a promising avenue for regional cardiac function analysis.

Imaging of cardiac perfusion with MR is a challenging area of research especially due to the motion of the heart and limited time of data acquisition. Compressed sensing is a popular signal estimation method recently adopted by researchers in MRI which can improve the spatial and/or temporal resolution of the acquired images by reducing the number of necessary samples for image reconstruction. This paper focuses on performance of temporal regularization with total variation and wavelets in compressed sensing. The impact of the choice of regularization parameters on the image quality and the temporal variation of intensity in region of interests (ROIs) are discussed. It is found that selecting the regularization parameter so as to optimize the quality of the reconstructed image sequence as a whole, leads to erroneous reconstruction of certain regions due to over regularization.

Tagged magnetic resonance imaging (tagged MRI or tMRI) provides a means of directly and noninvasively displaying the internal motion of the myocardium. Reconstruction of the motion field is needed to quantify important clinical information, e.g., the myocardial strain, and detect regional heart functional loss. In this paper, we present a three-step method for this task. First, we use a Gabor filter bank to detect and locate tag intersections in the image frames, based on local phase analysis. Next, we use an improved version of the robust point matching (RPM) method to sparsely track the motion of the myocardium, by establishing a transformation function and a one-to-one correspondence between grid tag intersections in different image frames. In particular, the RPM helps to minimize the impact on the motion tracking result of 1) through-plane motion and 2) relatively large deformation and/or relatively small tag spacing. In the final step, a meshless deformable model is initialized using the transformation function computed by RPM. The model refines the motion tracking and generates a dense displacement map, by deforming under the influence of image information, and is constrained by the displacement magnitude to retain its geometric structure. The 2D displacement maps in short and long axis image planes can be combined to drive a 3D deformable model, using the moving least square method, constrained by the minimization of the residual error at tag intersections. The method has been tested on a numerical phantom, as well as on in vivo heart data from normal volunteers and heart disease patients. The experimental results show that the new method has a good performance on both synthetic and real data. Furthermore, the method has been used in an initial clinical study to assess the differences in myocardial strain distributions between heart disease (left ventricular hypertrophy) patients and the normal control group. The final results show that the proposed method is capable of separating patients from healthy individuals. In addition, the method detects and makes possible quantification of local abnormalities in the myocardium strain distribution, which is critical for quantitative analysis of patients' clinical conditions. This motion tracking approach can improve the throughput and reliability of quantitative strain analysis of heart disease patients, and has the potential for further clinical applications.

We propose a novel meshless deformable model for in vivo cardiac left ventricle (LV) 3D motion estimation. As a relatively new technology, taggedMRI (tMRI) provides a direct and noninvasive way to reveal local deformation of the myocardium, which creates a large amount of heart motion data which requiring quantitative analysis. In our study, we sample the heart motion sparsely at intersections of three sets of orthogonal tagging planes and then use a new meshless deformable model to recover the dense 3D motion of the myocardium temporally during the cardiac cycle. We compute external forces at tag intersections based on tracked local motion and redistribute the force to meshless particles throughout the myocardium. Internal constraint forces at particles are derived from local strain energy using a Moving Least Squares (MLS) method. The dense 3D motion field is then computed and updated using the Lagrange equation. The new model avoids the singularity problem of mesh-based models and is capable of tracking large deformation with high efficiency and accuracy. In particular, the model performs well even when the control points (tag intersections) are relatively sparse. We tested the performance of the meshless model on a numerical phantom, as well as in vivo heart data of healthy subjects and patients. The experimental results show that the meshless deformable model can fully recover the myocardium motion in 3D.

The purposes of this study were to develop a semiautomated cardiac contour segmentation method for use with cine displacement-encoded MRI and evaluate its accuracy against manual segmentation. This segmentation model was designed with two distinct phases: preparation and evolution. During the model preparation phase, after manual image cropping and then image intensity standardization, the myocardium is separated from the background based on the difference in their intensity distributions, and the endo- and epi-cardial contours are initialized automatically as zeros of an underlying level set function. During the model evolution phase, the model deformation is driven by the minimization of an energy function consisting of five terms: model intensity, edge attraction, shape prior, contours interaction, and contour smoothness. The energy function is minimized iteratively by adaptively weighting the five terms in the energy function using an annealing algorithm. The validation experiments were performed on a pool of cine data sets of five volunteers. The difference between the semiautomated segmentation and manual segmentation was sufficiently small as to be considered clinically irrelevant. This relatively accurate semiautomated segmentation method can be used to significantly increase the throughput of strain analysis of cine displacement-encoded MR images for clinical applications.

Deformable models have had great successes over the past 20 years in medical applications. We have recently developed new classes of deformable models which we term hybrid deformable models to automate the model initialization process and make improvements in segmentation and registration. In this paper we present several hybrid deformable methods we have been developing for segmentation and registration. These methods include metamorphs, a novel shape and texture integration deformable model framework and the integration of deformable models with graphical models and learning methods. We first present a framework for the robust segmentation and tracking of the heart from tagged MRI images and second applications involving brain tumor segmentation as well as brain and cardiac shape registration

Heart ventricular mechanics has been investigated intensively in the last four decades. The passive material properties, the ventricular geometry and muscular architecture, and the myocardial activation are among the most important determinants of cardiac mechanics. The heart muscle is anisotropic, inhomogeneous, and highly nonlinear. The heart ventricular geometry is irregular and object dependent. The muscular architecture includes the organization of the fiber and the connective tissues. Studies of the myocardial activation have been carried out at both cell and tissue levels.Previous work from our research group has successfully estimated the in-vivo motion and deformation of both the left and the right ventricles. In this paper, we present an iterative model to estimate the in-vivo myocardium material properties, the active forces generated along fiber orientation, and strain and stress distribution in both ventricles. Compared to the strain energy function approach, our model is more intuitively understandable. Using the model, we have simulated the mechanical events of a few different heart diseases. Noticeable strain and stress differences are found between normal and diseased hearts.

The use of intracoronary artery stents in general medical care is increasing, as is the use of magnetic resonance imaging (MRI) scanning. We have developed software to model the risk of electrical generation from these intracoronary stents being placed in an MRI scanner's magnetic field, while the heart is beating. This has not been specifically modeled in the past. We developed a pair of programs to first measure the coronary artery movement, from cine-MRI scans of the heart; we used this data to calculate local voltages between the stents and adjacent myocardial tissue in various locations of the heart using a software model. The software is written in RealBasic and Python under Macintosh OS X.

Right ventricular (RV) dysfunction can serve as an indicator of heart and lung disease and can adversely affect the left ventricle. However, normal RV function must be characterized before abnormal states can be detected. We describe a method for reconstructing the 3D motion of the RV by fitting a deformable model to tag and contour data extracted from multiview tagged magnetic resonance images. The deformable model is a biventricular finite element mesh built directly from segmented contours. Our approach accommodates the geometrically complex RV by using the entire lengths of the tags, localized degrees of freedom, and finite elements for geometric modeling. Also, we outline methods for converting the 3D motion reconstruction results into potentially useful motion variables, such as strains and displacements. The technique was applied to synthetic data, two normal hearts, and two hearts with right ventricular hypertrophy (RVH). Noticeable differences were found between the motion variables calculated for normal volunteers and RVH patients.

We present a validation study of an optical-flow method for the rapid estimation of myocardial displacement in magnetic resonance tagged cardiac images. This registration and change visualization (RCV) software uses a hierarchical estimation technique to compute the flow field that describes the warping of an image of one cardiac phase into alignment with the next. This method overcomes the requirement of constant pixel intensity in standard optical-flow methods by preprocessing the input images to reduce any intensity bias which results from the reduction in stripe contrast throughout the cardiac cycle. To validate the method, SPAMM-tagged images were acquired of a silicon gel phantom with simulated rotational motion. The pixel displacement was estimated with the RCV method and the error in pixel tracking was <4% 1000 ms after application of the tags, and after 30 degrees of rotation. An additional study was performed using a SPAMM-tagged multiphase slice of a canine left ventricle. The true displacement was determined using a previously validated active contour model (snakes). The error between methods was 6.7% at end systole. The RCV method has the advantage of tracking all pixels in the image in a substantially shorter period than the snakes method.

We present a new approach for the analysis of the left ventricular shape and motion based on the development of a new class of volumetric deformable models. We estimate the deformation and complex motion of the left ventricle (LV) in terms of a few parameters that are functions and whose values vary locally across the LV. These parameters capture the radial and longitudinal contraction, the axial twisting, and the long-axis deformation. Using Lagrangian dynamics and finite-element theory, we convert these volumetric primitives into dynamic models that deform due to forces exerted by the datapoints. We present experiments where we used magnetic tagging (MRI-SPAMM) to acquire datapoints from the LV during systole. By applying our method to MRI-SPAMM datapoints, we were able to characterize the 3-D shape and motion of the LV both locally and globally, in a clinically useful way. In addition, based on the model parameters we were able to extract quantitative differences between normal and abnormal hearts and visualize them in a way that is useful to physicians.

The authors present a new method for analyzing the motion of the heart's left ventricle (LV) from tagged magnetic resonance imaging (MRI) data. Their technique is based on the development of a new class of physics-based deformable models whose parameters are functions. They allow the definition of new parameterized primitives and parameterized deformations which can capture the local shape variation of a complex object. Furthermore, these parameters are intuitive and require no complex post-processing in order to be used by a physician. Using a physics-based approach, the authors convert the geometric models into dynamic models that deform due to forces exerted from the datapoints and conform to the given dataset. The authors present experiments involving the extraction of the shape and motion of the LV's mid-wall during systole from tagged MRI data based on a few parameter functions. Furthermore, by plotting the variations over time of the extracted LV model parameters from normal and abnormal heart data along the long axis, the authors are able to quantitatively characterize their differences.

We present a new method for analyzing the 3D motion of the heart's left ventricle (LV) from tagged magnetic resonance imaging (MRI) data. Our technique is based on the development of a new class of volumetric physics-based deformable models whose parameters are functions and can capture the local shape variation of an object. These parameters require no complex post-processing in order to be used by a physician. These volumetric models allow the accurate estimation of the shape and motion of the inner and outer walls of the LV as well as within the walls. We also present a new technique for calculating forces exerted by tagged MRI data to material points of the deformable model. Furthermore, by plotting the variations over time of the extracted LV model parameters from normal heart data we are able to quantitatively analyze and compare the epicardial and endocardial motion.>

Tissue tagging using magnetic resonance (MR) imaging has enabled quantitative noninvasive analysis of motion and deformation in vivo. One method for MR tissue tagging is Spatial Modulation of Magnetization (SPAMM). Manual detection and tracking of tissue tags by visual inspection remains a time-consuming and tedious process. The authors have developed an interactively guided semi-automated method of detecting and tracking tag intersections in cardiac MR images. A template matching approach combined with a novel adaptation of active contour modeling permits rapid analysis of MR images. The authors have validated their technique using MR SPAMM images of a silicone gel phantom with controlled deformations. Average discrepancy between theoretically predicted and semi-automatically selected tag intersections was 0.30 mm+/-0.17 [mean+/-SD, NS (P<0.05)]. Cardiac SPAMM images of normal volunteers and diseased patients also have been evaluated using the authors' technique.

Magnetic resonance tissue tagging allows noninvasive in vivo measurement of soft tissue deformation. Planes of magnetic saturation are created, orthogonal to the imaging plane, which form dark lines (stripes) in the image. The authors describe a method for tracking stripe motion in the image plane, and show how this information can be incorporated into a finite element model of the underlying deformation. Human heart data were acquired from several imaging planes in different orientations and were combined using a deformable model of the left ventricle wall. Each tracked stripe point provided information on displacement orthogonal to the original tagging plane, i.e., a one-dimensional (1-D) constraint on the motion. Three-dimensional (3-D) motion and deformation was then reconstructed by fitting the model to the data constraints by linear least squares. The average root mean squared (rms) error between tracked stripe points and predicted model locations was 0.47 mm (n=3,100 points). In order to validate this method and quantify the errors involved, the authors applied it to images of a silicone gel phantom subjected to a known, well-controlled, 3-D deformation. The finite element strains obtained were compared to an analytic model of the deformation known to be accurate in the central axial plane of the phantom. The average rms errors were 6% in both the reconstructed shear strains and 16% in the reconstructed radial normal strain.

We develop a new method for analyzing the motion of the left ventricle (LV) of a heart from tagged MRI data. Our technique is based on the development of a new class of physics-based deformable models whose parameters are functions allowing the definition of new parameterized primitives and parameterized deformations. These parameter functions improve the accuracy of shape description through the use of a few intuitive parameters such as functional twisting. Furthermore, these parameters require no complex post-processing in order to be used by a physician. Using a physics-based approach, we convert these geometric models into deformable models that deform due to forces exerted from the datapoints and conform to the given dataset. We present experiments involving the extraction of shape and motion of the LV from MRI-SPAMM data based on a few parameter functions. Furthermore, by plotting the variations over time of the extracted model parameters from normal and abnormal heart data we are able to characterize quantitatively their differences.>

A measure of deformation energy suitable for fitting deformable models to image data is described. An object's displacement is constrained to be globally smooth by penalizing the variation of the deformation gradient tensor. This homogeneous deformation measure is invariant to arbitrary rigid body motion of object and viewpoint, given the correspondence between model and data. It remains quadratic in the displacement parameters, leading to linear-least-squares fits. The method was used to reconstruct the nonhomogeneous 3-D motion of the heart wall from tomographic magnetic resonance images. A finite-element model of the left ventricle was deformed to fit material points tracked in biplanar views. Only the in-plane components were available from each separate image, the through-plane components being reconstructed in the fit.>