Biomechanical modelling of soft tissue provides a method for constraining medical image registration, such that the estimated spatial transformation is considered biophysically plausible. Existing methods either directly optimize the loss function containing the biomechanical-constrained regularization term over deformations, which takes much computational time, or are trained using biomechanically plausible data generated via finite element simulation, which is cumbersome. This work first instantiates the recently-proposed physics-informed neural networks (PINNs) to 3D elastic models that are used to establish the partial differential equations (PDEs) representing physics laws of biomechanical constraints to be satisfied. The registration algorithm that aligns point sets considering PINN-imposed biomechanics (i.e., the forward problem) is then formulated. In addition, the inverse problem and its algorithm of physical parameter (i.e., material property) estimation along with the registration are also formulated and developed. We carefully compare linear and nonlinear elasticity theories' capabilities in solving both tasks of forward registration and inverse physical parameter identification under PINNs respectively. Furthermore, two specific network configurations that leverage one common branch or two individual branches to predict deformation vectors and biomechanical states are also constructed and compared. The proposed PINNs-based registration approaches have been extensively evaluated with three experiments, that is single and multiple patient MRI-US registration using clinical MRI-US pairs, and registration using pairs of undeformed MR images from clinical cases of prostate cancer biopsy and deformed counterparts with finite-element-computed ground-truth deformation. Results demonstrate that the proposed methods achieve state-of-the-art performances compared to biomechanical-model-based and learning-based registration approaches, and the biomechanical constraints of soft tissues have been successfully warranted after registration. The codes are available at https://github.com/ZheMin-1992/Registration_PINNs.

Minimally invasive surgery (MIS) remains technically demanding due to the difficulty of tracking hidden critical structures within the moving anatomy of the patient. In this study, we propose a soft tissue deformation tracking augmented reality (AR) navigation pipeline for laparoscopic surgery of the kidneys. The proposed navigation pipeline addresses two main sub-problems: the initial registration and deformation tracking. Our method utilizes preoperative MR or CT data and binocular laparoscopes without any additional interventional hardware. The initial registration is resolved through a probabilistic rigid registration algorithm and elastic compensation based on dense point cloud reconstruction. For deformation tracking, the sparse feature point displacement vector field continuously provides temporal boundary conditions for the biomechanical model. To enhance the accuracy of the displacement vector field, a novel feature points selection strategy based on deep learning is proposed. Moreover, an ex-vivo experimental method for internal structures error assessment is presented. The ex-vivo experiments indicate an external surface reprojection error of 4.07 ± 2.17 mm and a maximum mean absolutely error for internal structures of 2.98 mm. In-vivo experiments indicate mean absolutely error of 3.28 ± 0.40 mm and 1.90 ± 0.24 mm, respectively. The combined qualitative and quantitative findings indicated the potential of our AR-assisted navigation system in improving the clinical application of laparoscopic kidney surgery.

Deformable image registration (DIR) can be used to track cardiac motion. Conventional DIR algorithms aim to establish a dense and non-linear correspondence between independent pairs of images. They are, nevertheless, computationally intensive and do not consider temporal dependencies to regulate the estimated motion in a cardiac cycle. In this paper, leveraging deep learning methods, we formulate a novel hierarchical probabilistic model, termed DragNet, for fast and reliable spatio-temporal registration in cine cardiac magnetic resonance (CMR) images and for generating synthetic heart motion sequences. DragNet is a variational inference framework, which takes an image from the sequence in combination with the hidden states of a recurrent neural network (RNN) as inputs to an inference network per time step. As part of this framework, we condition the prior probability of the latent variables on the hidden states of the RNN utilised to capture temporal dependencies. We further condition the posterior of the motion field on a latent variable from hierarchy and features from the moving image. Subsequently, the RNN updates the hidden state variables based on the feature maps of the fixed image and the latent variables. Different from traditional methods, DragNet performs registration on unseen sequences in a forward pass, which significantly expedites the registration process. Besides, DragNet enables generating a large number of realistic synthetic image sequences given only one frame, where the corresponding deformations are also retrieved. The probabilistic framework allows for computing spatio-temporal uncertainties in the estimated motion fields. Our results show that DragNet performance is comparable with state-of-the-art methods in terms of registration accuracy, with the advantage of offering analytical pixel-wise motion uncertainty estimation across a cardiac cycle and being a motion generator. We will make our code publicly available.

A probabilistic group-wise similarity registration technique based on Student's t-mixture model (TMM) and a multi-resolution extension of the same (mr-TMM) are proposed in this study, to robustly align shapes and establish valid correspondences, for the purpose of training statistical shape models (SSMs). Shape analysis across large cohorts requires automatic generation of the requisite training sets. Automated segmentation and landmarking of medical images often result in shapes with varying proportions of outliers and consequently require a robust method of alignment and correspondence estimation. Both TMM and mrTMM are validated by comparison with state-of-the-art registration algorithms based on Gaussian mixture models (GMMs), using both synthetic and clinical data. Four clinical data sets are used for validation: (a) 2D femoral heads (K= 1000 samples generated from DXA images of healthy subjects); (b) control-hippocampi (K= 50 samples generated from T1-weighted magnetic resonance (MR) images of healthy subjects); (c) MCI-hippocampi (K= 28 samples generated from MR images of patients diagnosed with mild cognitive impairment); and (d) heart shapes comprising left and right ventricular endocardium and epicardium (K= 30 samples generated from short-axis MR images of: 10 healthy subjects, 10 patients diagnosed with pulmonary hypertension and 10 diagnosed with hypertrophic cardiomyopathy). The proposed methods significantly outperformed the state-of-the-art in terms of registration accuracy in the experiments involving synthetic data, with mrTMM offering significant improvement over TMM. With the clinical data, both methods performed comparably to the state-of-the-art for the hippocampi and heart data sets, which contained few outliers. They outperformed the state-of-the-art for the femur data set, containing large proportions of outliers, in terms of alignment accuracy, and the quality of SSMs trained, quantified in terms of generalization, compactness and specificity.

Three-dimensional reconstructions of coronary arteries can overcome some of the limitations of 2D X-ray angiography, namely artery overlap/foreshortening and lack of depth information. Model-based arterial reconstruction algorithms usually rely on 2D coronary artery segmentations and require good robustness to outliers. In this paper, we propose a novel probabilistic method to reconstruct coronary artery centrelines from retrospectively gated X-ray images based on a probabilistic mixture model. Specifically, 3D coronary artery centrelines are described by a mixture of Student’s t-distributions, and the reconstruction is formulated as maximum-likelihood estimation of the mixture model parameters, given the 2D segmentations of arteries from 2D X-ray images. Our method provides robustness against the erroneously segmented parts in the 2D segmentations by taking advantage of the inherent robustness of t-distributions. We validate our reconstruction results using synthetic phantom and clinical X-ray angiography data. The results show that the proposed method can cope with imperfect and noisy segmentation data.

A novel probabilistic, group-wise rigid registration framework is proposed in this study, to robustly align and establish correspondence across anatomical shapes represented as unstructured point sets. Student’s t-mixture model (TMM) is employed to exploit their inherent robustness to outliers. The primary application for such a framework is the automatic construction of statistical shape models (SSMs) of anatomical structures, from medical images. Tools used for automatic segmentation and landmarking of medical images often result in segmentations with varying proportions of outliers. The proposed approach is able to robustly align shapes and establish valid correspondences in the presence of considerable outliers and large variations in shape. A multi-resolution registration (mrTMM) framework is also formulated, to further improve the performance of the proposed TMM-based registration method. Comparisons with a state-of-the art approach using clinical data show that the mrTMM method in particular, achieves higher alignment accuracy and yields SSMs that generalise better to unseen shapes.

Spine-related disorders are amongst the most frequently encountered problems in clinical medicine. For several applications such as 1) to improve the assessment of the strength of the spine, as well as 2) to optimize the personalization of spinal interventions, image-based biomechanical modeling of the vertebrae is expected to play an important predictive role. However, this requires the construction of computational models that are subject-specific and comprehensive. In particular, they need to incorporate information about the vertebral anisotropic micro-architecture, which plays a central role in the biomechanical function of the vertebrae. In practice, however, accurate personalization of the vertebral trabeculae has proven to be difficult as its imaging in vivo is currently infeasible. Consequently, this paper presents a statistical approach for accurate prediction of the vertebral fabric tensors based on a training sample of ex vivo micro-CT images. To the best of our knowledge, this is the first predictive model proposed and validated for vertebral datasets. The method combines features selection and partial least squares regression in order to derive optimal latent variables for the prediction of the fabric tensors based on the more easily extracted shape and density information. Detailed validation with 20 ex vivo T12 vertebrae demonstrates the accuracy and consistency of the approach for the personalization of trabecular anisotropy.

Determining corresponding regions between an MRI and an X-ray mammogram is a clinically useful task that is challenging for radiologists due to the large deformation that the breast undergoes between the two image acquisitions. In this work we propose an intensity-based image registration framework, where the biomechanical transformation model parameters and the rigid-body transformation parameters are optimised simultaneously. Patient-specific biomechanical modelling of the breast derived from diagnostic, prone MRI has been previously used for this task. However, the high computational time associated with breast compression simulation using commercial packages, did not allow the optimisation of both pose and FEM parameters in the same framework. We use a fast explicit Finite Element (FE) solver that runs on a graphics card, enabling the FEM-based transformation model to be fully integrated into the optimisation scheme. The transformation model has seven degrees of freedom, which include parameters for both the initial rigid-body pose of the breast prior to mammographic compression, and those of the biomechanical model. The framework was tested on ten clinical cases and the results were compared against an affine transformation model, previously proposed for the same task. The mean registration error was 11.6±3.8mm for the CC and 11±5.4mm for the MLO view registrations, indicating that this could be a useful clinical tool.

Reduced order modelling, in which a full system response is projected onto a subspace of lower dimensionality, has been used previously to accelerate finite element solution schemes by reducing the size of the involved linear systems. In the present work we take advantage of a secondary effect of such reduction for explicit analyses, namely that the stable integration time step is increased far beyond that of the full system. This phenomenon alleviates one of the principal drawbacks of explicit methods, compared with implicit schemes. We present an explicit finite element scheme in which time integration is performed in a reduced basis. Futhermore, we present a simple procedure for imposing inhomogeneous essential boundary conditions, thus overcoming one of the principal deficiencies of such approaches. The computational benefits of the procedure within a GPU-based execution framework are examined, and an assessment of the errors introduced is given. It is shown that speedups approaching an order of magnitude are feasible, without introduction of prohibitive errors, and without hardware modifications. The procedure may have applications in interactive simulation and medical image-guidance problems, in which both speed and accuracy are vital.

The use of biomechanical modelling, especially in conjunction with finite element analysis, has become common in many areas of medical image analysis and surgical simulation. Clinical employment of such techniques is hindered by conflicting requirements for high fidelity in the modelling approach, and fast solution speeds. We report the development of techniques for high-speed nonlinear finite element analysis for surgical simulation. We use a fully nonlinear total Lagrangian explicit finite element formulation which offers significant computational advantages for soft tissue simulation. However, the key contribution of the work is the presentation of a fast graphics processing unit (GPU) solution scheme for the finite element equations. To the best of our knowledge, this represents the first GPU implementation of a nonlinear finite element solver. We show that the present explicit finite element scheme is well suited to solution via highly parallel graphics hardware, and that even a midrange GPU allows significant solution speed gains (up to 16.8 x) compared with equivalent CPU implementations. For the models tested the scheme allows real-time solution of models with up to 16,000 tetrahedral elements. The use of GPUs for such purposes offers a cost-effective high-performance alternative to expensive multi-CPU machines, and may have important applications in medical image analysis and surgical simulation.