Diffusion models (DMs) have recently been introduced as a regularizing prior for PET image reconstruction, integrating DMs trained on high-quality PET images with unsupervised schemes that condition on measured data. While these approaches have potential generalization advantages due to their independence from the scanner geometry and the injected activity level, they forgo the opportunity to explicitly model the interaction between the DM prior and noisy measurement data, potentially limiting reconstruction accuracy. To address this, we propose a supervised DM-based algorithm for PET reconstruction. Our method enforces the non-negativity of PET’s Poisson likelihood model and accommodates the wide intensity range of PET images. Through experiments on realistic brain PET phantoms, we demonstrate that our approach outperforms or matches state-of-the-art deep learning-based methods quantitatively across a range of dose levels. We further conduct ablation studies to demonstrate the benefits of the proposed components in our model, as well as its dependence on training data, parameter count, and number of diffusion steps. Additionally, we show that our approach enables more accurate posterior sampling than unsupervised DM-based methods, suggesting improved uncertainty estimation. Finally, we extend our methodology to a practical approach for fully 3D PET and present example results from real [ $$^{18}$$ F]FDG brain PET data.

Splenomegaly, the enlargement of the spleen, is an important clinical indicator for various associated medical conditions, such as sickle cell disease (SCD). Spleen length measured from 2D ultrasound is the most widely used metric for characterising spleen size. However, it is still considered a surrogate measure, and spleen volume remains the gold standard for assessing spleen size. Accurate spleen volume measurement typically requires 3D imaging modalities, such as computed tomography or magnetic resonance imaging, but these are not widely available, especially in the Global South which has a high prevalence of SCD. In this work, we introduce a deep learning pipeline, DeepSPV, for precise spleen volume estimation from single or dual 2D ultrasound images. The pipeline involves a segmentation network and a variational autoencoder for learning low-dimensional representations from the estimated segmentations. We investigate three approaches for spleen volume estimation and our best model achieves 86.62%/92.5% mean relative volume accuracy (MRVA) under single-view/dual-view settings, surpassing the performance of human experts. In addition, the pipeline can provide confidence intervals for the volume estimates as well as offering benefits in terms of interpretability, which further support clinicians in decision-making when identifying splenomegaly. We evaluate the full pipeline using a highly realistic synthetic dataset generated by a diffusion model, achieving an overall MRVA of 83.0% from a single 2D ultrasound image. Our proposed DeepSPV is the first work to use deep learning to estimate 3D spleen volume from 2D ultrasound images and can be seamlessly integrated into the current clinical workflow for spleen assessment. We also make our synthetic spleen ultrasound dataset publicly available.

Medical image reconstruction with pre-trained score-based generative models (SGMs) has advantages over other existing state-of-the-art deep-learned reconstruction methods, including improved resilience to different scanner setups and advanced image distribution modeling. SGM-based reconstruction has recently been applied to simulated positron emission tomography (PET) datasets, showing improved contrast recovery for out-of-distribution lesions relative to the state-of-the-art. However, existing methods for SGM-based reconstruction from PET data suffer from slow reconstruction, burdensome hyperparameter tuning and slice inconsistency effects (in 3D). In this work, we propose a practical methodology for fully 3D reconstruction that accelerates reconstruction and reduces the number of critical hyperparameters by matching the likelihood of an SGM's reverse diffusion process to a current iterate of the maximum-likelihood expectation maximization algorithm. Using the example of low-count reconstruction from simulated [ ${}^{{18}}$ F]DPA-714 datasets, we show our methodology can match or improve on the NRMSE and SSIM of existing state-of-the-art SGM-based PET reconstruction while reducing reconstruction time and the need for hyperparameter tuning. We evaluate our methodology against state-of-the-art supervised and conventional reconstruction algorithms. Finally, we demonstrate a first-ever implementation of SGM-based reconstruction for real 3D PET data, specifically [ ${}^{{18}}$ F]DPA-714 data, where we integrate perpendicular pre-trained SGMs to eliminate slice inconsistency issues.

Quantifying uncertainty of predictions has been identified as one way to develop more trustworthy artificial intelligence (AI) models beyond conventional reporting of performance metrics. When considering their role in a clinical decision support setting, AI classification models should ideally avoid confident wrong predictions and maximise the confidence of correct predictions. Models that do this are said to be well calibrated with regard to confidence. However, relatively little attention has been paid to how to improve calibration when training these models, i.e. to make the training strategy uncertainty-aware. In this work we: (i) evaluate three novel uncertainty-aware training strategies with regard to a range of accuracy and calibration performance measures, comparing against two state-of-the-art approaches, (ii) quantify the data (aleatoric) and model (epistemic) uncertainty of all models and (iii) evaluate the impact of using a model calibration measure for model selection in uncertainty-aware training, in contrast to the normal accuracy-based measures. We perform our analysis using two different clinical applications: cardiac resynchronisation therapy (CRT) response prediction and coronary artery disease (CAD) diagnosis from cardiac magnetic resonance (CMR) images. The best-performing model in terms of both classification accuracy and the most common calibration measure, expected calibration error (ECE) was the Confidence Weight method, a novel approach that weights the loss of samples to explicitly penalise confident incorrect predictions. The method reduced the ECE by 17% for CRT response prediction and by 22% for CAD diagnosis when compared to a baseline classifier in which no uncertainty-aware strategy was included. In both applications, as well as reducing the ECE there was a slight increase in accuracy from 69% to 70% and 70% to 72% for CRT response prediction and CAD diagnosis respectively. However, our analysis showed a lack of consistency in terms of optimal models when using different calibration measures. This indicates the need for careful consideration of performance metrics when training and selecting models for complex high risk applications in healthcare.

Multi-class segmentation of cardiac magnetic resonance (CMR) images seeks a separation of data into anatomical components with known structure and configuration. The most popular CNN-based methods are optimised using pixel wise loss functions, ignorant of the spatially extended features that characterise anatomy. Therefore, whilst sharing a high spatial overlap with the ground truth, inferred CNN-based segmentations can lack coherence, including spurious connected components, holes and voids. Such results are implausible, violating anticipated anatomical topology. In response, (single-class) persistent homology-based loss functions have been proposed to capture global anatomical features. Our work extends these approaches to the task of multi-class segmentation. Building an enriched topological description of all class labels and class label pairs, our loss functions make predictable and statistically significant improvements in segmentation topology using a CNN-based post-processing framework. We also present (and make available) a highly efficient implementation based on cubical complexes and parallel execution, enabling practical application within high resolution 3D data for the first time. We demonstrate our approach on 2D short axis and 3D whole heart CMR segmentation, advancing a detailed and faithful analysis of performance on two publicly available datasets.

We present a novel multimodal deep learning framework for cardiac resynchronisation therapy (CRT) response prediction from 2D echocardiography and cardiac magnetic resonance (CMR) data. The proposed method first uses the 'nnU-Net' segmentation model to extract segmentations of the heart over the full cardiac cycle from the two modalities. Next, a multimodal deep learning classifier is used for CRT response prediction, which combines the latent spaces of the segmentation models of the two modalities. At test time, this framework can be used with 2D echocardiography data only, whilst taking advantage of the implicit relationship between CMR and echocardiography features learnt from the model. We evaluate our pipeline on a cohort of 50 CRT patients for whom paired echocardiography/CMR data were available, and results show that the proposed multimodal classifier results in a statistically significant improvement in accuracy compared to the baseline approach that uses only 2D echocardiography data. The combination of multimodal data enables CRT response to be predicted with 77.38% accuracy (83.33% sensitivity and 71.43% specificity), which is comparable with the current state-of-the-art in machine learning-based CRT response prediction. Our work represents the first multimodal deep learning approach for CRT response prediction.

We introduce a method for training neural networks to perform image or volume segmentation in which prior knowledge about the topology of the segmented object can be explicitly provided and then incorporated into the training process. By using the differentiable properties of persistent homology, a concept used in topological data analysis, we can specify the desired topology of segmented objects in terms of their Betti numbers and then drive the proposed segmentations to contain the specified topological features. Importantly this process does not require any ground-truth labels, just prior knowledge of the topology of the structure being segmented. We demonstrate our approach in four experiments: one on MNIST image denoising and digit recognition, one on left ventricular myocardium segmentation from magnetic resonance imaging data from the UK Biobank, one on the ACDC public challenge dataset and one on placenta segmentation from 3-D ultrasound. We find that embedding explicit prior knowledge in neural network segmentation tasks is most beneficial when the segmentation task is especially challenging and that it can be used in either a semi-supervised or post-processing context to extract a useful training gradient from images without pixelwise labels.

Manifold alignment (MA) is a technique to map many high-dimensional datasets to one shared low-dimensional space. Here we develop a pipeline for using MA to reconstruct high-resolution medical images. We present two key contributions. First, we develop a novel MA scheme in which each high-dimensional dataset can be differently weighted preventing noisier or less informative data from corrupting the aligned embedding. We find that this generalisation improves performance in our experiments in both supervised and unsupervised MA problems. Second, we use the wave kernel signature as a graph descriptor for the unsupervised MA case finding that it significantly outperforms the current state-of-the-art methods and provides higher quality reconstructed magnetic resonance volumes than existing methods.

Segmenting anatomical structures in medical images has been successfully addressed with deep learning methods for a range of applications. However, this success is heavily dependent on the quality of the image that is being segmented. A commonly neglected point in the medical image analysis community is the vast amount of clinical images that have severe image artefacts due to organ motion, movement of the patient and/or image acquisition related issues. In this paper, we discuss the implications of image motion artefacts on cardiac MR segmentation and compare a variety of approaches for jointly correcting for artefacts and segmenting the cardiac cavity. The method is based on our recently developed joint artefact detection and reconstruction method, which reconstructs high quality MR images from k-space using a joint loss function and essentially converts the artefact correction task to an under-sampled image reconstruction task by enforcing a data consistency term. In this paper, we propose to use a segmentation network coupled with this in an end-to-end framework. Our training optimises three different tasks: 1) image artefact detection, 2) artefact correction and 3) image segmentation. We train the reconstruction network to automatically correct for motion-related artefacts using synthetically corrupted cardiac MR k-space data and uncorrected reconstructed images. Using a test set of 500 2D+time cine MR acquisitions from the UK Biobank data set, we achieve demonstrably good image quality and high segmentation accuracy in the presence of synthetic motion artefacts. We showcase better performance compared to various image correction architectures.

Good quality of medical images is a prerequisite for the success of subsequent image analysis pipelines. Quality assessment of medical images is therefore an essential activity and for large population studies such as the UK Biobank (UKBB), manual identification of artefacts such as those caused by unanticipated motion is tedious and time-consuming. Therefore, there is an urgent need for automatic image quality assessment techniques. In this paper, we propose a method to automatically detect the presence of motion-related artefacts in cardiac magnetic resonance (CMR) cine images. We compare two deep learning architectures to classify poor quality CMR images: 1) 3D spatio-temporal Convolutional Neural Networks (3D-CNN), 2) Long-term Recurrent Convolutional Network (LRCN). Though in real clinical setup motion artefacts are common, high-quality imaging of UKBB, which comprises cross-sectional population data of volunteers who do not necessarily have health problems creates a highly imbalanced classification problem. Due to the high number of good quality images compared to the relatively low number of images with motion artefacts, we propose a novel data augmentation scheme based on synthetic artefact creation in k-space. We also investigate a learning approach using a predetermined curriculum based on synthetic artefact severity. We evaluate our pipeline on a subset of the UK Biobank data set consisting of 3510 CMR images. The LRCN architecture outperformed the 3D-CNN architecture and was able to detect 2D+time short axis images with motion artefacts in less than 1ms with high recall. We compare our approach to a range of state-of-the-art quality assessment methods. The novel data augmentation and curriculum learning approaches both improved classification performance achieving overall area under the ROC curve of 0.89.

Abnormal cardiac motion can indicate different forms of disease, which can manifest at different spatial scales in the myocardium. Many studies have sought to characterise particular motion abnormalities associated with specific diseases, and to utilise motion information to improve diagnoses. However, the importance of spatial scale in the analysis of cardiac deformation has not been extensively investigated. We build on recent work on the analysis of myocardial strains at different spatial scales using a cardiac motion atlas to find the optimal scales for estimating different cardiac biomarkers. We apply a multi-scale strain analysis to a 43 patient cohort of cardiac resynchronisation therapy (CRT) patients using tagged magnetic resonance imaging data for (1) predicting response to CRT, (2) identifying septal flash, (3) estimating QRS duration, and (4) identifying the presence of ischaemia. A repeated, stratified cross-validation is used to demonstrate the importance of spatial scale in our analysis, revealing different optimal spatial scales for the estimation of different biomarkers.

Respiratory motion poses significant challenges in image-guided interventions. In emerging treatments such as MR-guided HIFU or MR-guided radiotherapy, it may cause significant misalignments between interventional road maps obtained pre-procedure and the anatomy during the treatment, and may affect intra-procedural imaging such as MR-thermometry. Patient specific respiratory motion models provide a solution to this problem. They establish a correspondence between the patient motion and simpler surrogate data which can be acquired easily during the treatment. Patient motion can then be estimated during the treatment by acquiring only the simpler surrogate data. In the majority of classical motion modelling approaches once the correspondence between the surrogate data and the patient motion is established it cannot be changed unless the model is recalibrated. However, breathing patterns are known to significantly change in the time frame of MR-guided interventions. Thus, the classical motion modelling approach may yield inaccurate motion estimations when the relation between the motion and the surrogate data changes over the duration of the treatment and frequent recalibration may not be feasible. We propose a novel methodology for motion modelling which has the ability to automatically adapt to new breathing patterns. This is achieved by choosing the surrogate data in such a way that it can be used to estimate the current motion in 3D as well as to update the motion model. In particular, in this work, we use 2D MR slices from different slice positions to build as well as to apply the motion model. We implemented such an autoadaptive motion model by extending our previous work on manifold alignment. We demonstrate a proof-of-principle of the proposed technique on cardiac gated data of the thorax and evaluate its adaptive behaviour on realistic synthetic data containing two breathing types generated from 6 volunteers, and real data from 4 volunteers. On synthetic data the autoadaptive motion model yielded 21.45% more accurate motion estimations compared to a non-adaptive motion model 10 min after a change in breathing pattern. On real data we demonstrated the method's ability to maintain motion estimation accuracy despite a drift in the respiratory baseline. Due to the cardiac gating of the imaging data, the method is currently limited to one update per heart beat and the calibration requires approximately 12 min of scanning. Furthermore, the method has a prediction latency of 800 ms. These limitations may be overcome in future work by altering the acquisition protocol.

We present a framework for combining a cardiac motion atlas with non-motion data. The atlas represents cardiac cycle motion across a number of subjects in a common space based on rich motion descriptors capturing 3D displacement, velocity, strain and strain rate. The non-motion data are derived from a variety of sources such as imaging, electrocardiogram (ECG) and clinical reports. Once in the atlas space, we apply a novel supervised learning approach based on random projections and ensemble learning to learn the relationship between the atlas data and some desired clinical output. We apply our framework to the problem of predicting response to Cardiac Resynchronisation Therapy (CRT). Using a cohort of 34 patients selected for CRT using conventional criteria, results show that the combination of motion and non-motion data enables CRT response to be predicted with 91.2% accuracy (100% sensitivity and 62.5% specificity), which compares favourably with the current state-of-the-art in CRT response prediction.

Cardiac motion atlases provide a space of reference in which the motions of a cohort of subjects can be directly compared. Motion atlases can be used to learn descriptors that are linked to different pathologies and which can subsequently be used for diagnosis. To date, all such atlases have been formed and applied using data from the same modality. In this work we propose a framework to build a multimodal cardiac motion atlas from 3D magnetic resonance (MR) and 3D ultrasound (US) data. Such an atlas will benefit from the complementary motion features derived from the two modalities, and furthermore, it could be applied in clinics to detect cardiovascular disease using US data alone. The processing pipeline for the formation of the multimodal motion atlas initially involves spatial and temporal normalisation of subjects' cardiac geometry and motion. This step was accomplished following a similar pipeline to that proposed for single modality atlas formation. The main novelty of this paper lies in the use of a multi-view algorithm to simultaneously reduce the dimensionality of both the MR and US derived motion data in order to find a common space between both modalities to model their variability. Three different dimensionality reduction algorithms were investigated: principal component analysis, canonical correlation analysis and partial least squares regression (PLS). A leave-one-out cross validation on a multimodal data set of 50 volunteers was employed to quantify the accuracy of the three algorithms. Results show that PLS resulted in the lowest errors, with a reconstruction error of less than 2.3 mm for MR-derived motion data, and less than 2.5 mm for US-derived motion data. In addition, 1000 subjects from the UK Biobank database were used to build a large scale monomodal data set for a systematic validation of the proposed algorithms. Our results demonstrate the feasibility of using US data alone to analyse cardiac function based on a multimodal motion atlas.

We present a novel retrospective self-gating method based on manifold alignment (MA), which enables reconstruction of free breathing, high spatial, and temporal resolution abdominal magnetic resonance imaging sequences. Based on a radial golden-angle acquisition trajectory, our method enables a multidimensional self-gating signal to be extracted from the k -space data for more accurate motion representation. The k -space radial profiles are evenly divided into a number of overlapping groups based on their radial angles. MA is then used to simultaneously learn and align the low dimensional manifolds of all groups, and embed them into a common manifold. In the manifold, k -space profiles that represent similar respiratory positions are close to each other. Image reconstruction is performed by combining radial profiles with evenly distributed angles that are close in the manifold. Our method was evaluated on both 2-D and 3-D synthetic and in vivo data sets. On the synthetic data sets, our method achieved high correlation with the ground truth in terms of image intensity and virtual navigator values. Using the in vivo data, compared with a state-of-the-art approach based on the center of k -space gating, our method was able to make use of much richer profile data for self-gating, resulting in statistically significantly better quantitative measurements in terms of organ sharpness and image gradient entropy.

We present a novel framework for retrospective dynamic 3D volume reconstruction from a multi-slice MRI acquisition using manifold alignment. K-space data are continuously acquired under free breathing using a radial golden-angle trajectory in a slice-by-slice manner. Non-overlapping consecutive profiles that were acquired within a short time window are grouped together. All grouped profiles from all slices are then simultaneously embedded using manifold alignment into a common manifold space (MS), in which profiles that were acquired at similar respiratory states are close together. Subsequently, a 3D volume can be reconstructed at each of the grouped profile MS positions by combining profiles that are close in the MS. This enables the original multi-slice dataset to be used to reconstruct a dynamic 3D sequence based on the respiratory state correspondences established in the MS. Our method was evaluated on both synthetic and in vivo datasets. For the synthetic datasets, the reconstructed dynamic sequence achieved a normalised cross correlation of 0.98 and peak signal to noise ratio of 26.64 dB compared with the ground truth. For the in vivo datasets, based on sharpness measurements and visual comparison, our method performed better than reconstruction using an adapted central k-space gating method.

Magnetic resonance imaging (MRI) has been commonly used for guiding and planning image guided interventions since it provides excellent soft tissue visualization of anatomy and allows motion modeling to predict the position of target tissues during the procedure. However, MRI-based motion modeling remains challenging due to the difficulty of acquiring multiple motion-free 3-D respiratory phases with adequate contrast and spatial resolution. Here, we propose a novel retrospective respiratory gating scheme from a 3-D undersampled high-resolution MRI acquisition combined with fast and robust image registrations to model the nonrigid deformation of the liver. The acquisition takes advantage of the recently introduced golden-radial phase encoding (G-RPE) trajectory. G-RPE is self-gated, i.e., the respiratory signal can be derived from the acquired data itself, and allows retrospective reconstructions of multiple respiratory phases at any arbitrary respiratory position. Nonrigid motion modeling is applied to predict the liver deformation of an average breathing cycle. The proposed approach was validated on 10 healthy volunteers. Motion model accuracy was assessed using similarity-, surface-, and landmark-based validation methods, demonstrating precise model predictions with an overall target registration error of TRE = 1.70 ± 0.94 mm which is within the range of the acquired resolution.

For many image-guided interventions there exists a need to compute the registration between preprocedure image(s) and the physical space of the intervention. Real-time intraprocedure imaging such as ultrasound (US) can be used to image the region of interest directly and provide valuable anatomical information for computing this registration. Unfortunately, real-time US images often have poor signal-to-noise ratio and suffer from imaging artefacts. Therefore, registration using US images can be challenging and significant preprocessing is often required to make the registrations robust. In this paper we present a novel technique for computing the image-to-physical registration for minimally invasive cardiac interventions using 3-D US. Our technique uses knowledge of the physics of the US imaging process to reduce the amount of preprocessing required on the 3-D US images. To account for the fact that clinical US images normally undergo significant image processing before being exported from the US machine our optimization scheme allows the parameters of the US imaging model to vary. We validated our technique by computing rigid registrations for 12 cardiac US/magnetic resonance imaging (MRI) datasets acquired from six volunteers and two patients. The technique had mean registration errors of 2.1-4.4 mm, and 75% capture ranges of 5-30 mm. We also demonstrate how the same approach can be used for respiratory motion correction: on 15 datasets acquired from five volunteers the registration errors due to respiratory motion were reduced by 45%-92%.

This paper describes a predictive and adaptive single parameter motion model for updating roadmaps to correct for respiratory motion in image-guided interventions. The model can adapt its motion estimates to respond to changes in breathing pattern, such as deep or fast breathing, which normally would result in a decrease in the accuracy of the motion estimates. The adaptation is made possible by interpolating between the motion estimates of multiple submodels, each of which describes the motion of the target organ during cycles of different amplitudes. We describe a predictive technique which can predict the amplitude of a breathing cycle before it has finished. The predicted amplitude is used to interpolate between the motion estimates of the submodels to tune the adaptive model to the current breathing pattern. The proposed technique is validated on affine motion models formed from cardiac magnetic resonance imaging (MRI) datasets acquired from seven volunteers and one patient. The amplitude prediction technique showed errors of 1.9-6.5 mm. The combined predictive and adaptive technique showed 3-D motion prediction errors of 1.0-2.8 mm, which represents an improvement in modelling performance of up to 40% over a standard nonadaptive single parameter motion model. We also applied the combined technique in a clinical setting to test the feasibility of using it for respiratory motion correction of roadmaps in image-guided cardiac catheterisations. In this clinical case we show that 2-D registration errors due to respiratory motion are reduced from 7.7 to 2.8 mm using the proposed technique.

We present a method for alignment of an interventional plan to optically tracked two-dimensional intraoperative ultrasound (US) images of the liver. Our clinical motivation is to enable the accurate transfer of information from three-dimensional preoperative imaging modalities [magnetic resonance (MR) or computed tomography (CT)] to intraoperative US to aid needle placement for thermal ablation of liver metastases. An initial rigid registration to intraoperative coordinates is obtained using a set of US images acquired at maximum exhalation. A preprocessing step is applied to both the preoperative images and the US images to produce evidence of corresponding structures. This yields two sets of images representing classification of regions as vessels. The registration then proceeds using these images. The preoperative images and plan are then warped to correspond to a single US slice acquired at an unknown point in the breathing cycle where the liver is likely to have moved and deformed relative to the preoperative image. Alignment is constrained using a patient-specific model of breathing motion and deformation. Target registration error is estimated by carrying out simulation experiments using resliced MR volumes to simulate real US and comparing the registration results to a "bronze-standard" registration performed on the full MR volume. Finally, the system is tested using real US and verified using visual inspection.

The problem of providing surgical navigation using image overlays on the operative scene can be split into four main tasks--calibration of the optical system; registration of preoperative images to the patient; system and patient tracking, and display using a suitable visualization scheme. To achieve a convincing result in the magnified microscope view a very high alignment accuracy is required. We have simulated an entire image overlay system to establish the most significant sources of error and improved each of the stages involved. The microscope calibration process has been automated. We have introduced bone-implanted markers for registration and incorporated a locking acrylic dental stent (LADS) for patient tracking. The LADS can also provide a less-invasive registration device with mean target error of 0.7 mm in volunteer experiments. These improvements have significantly increased the alignment accuracy of our overlays. Phantom accuracy is 0.3-0.5 mm and clinical overlay errors were 0.5-1.0 mm on the bone fiducials and 0.5-4 mm on target structures. We have improved the graphical representation of the stereo overlays. The resulting system provides three-dimensional surgical navigation for microscope-assisted guided interventions (MAGI).