Scatterplots are widely used in Explainable Artificial Intelligence (XAI) to investigate misclassifications and patterns across instances. However, a significant limitation of scatterplots is overplotting, especially when working with large datasets. Although several quality metrics have been proposed to measure the degree of overplotting, none have been demonstrated to be effective in the context of XAI. This paper aims to evaluate the effectiveness of a quality metric, called OM4AnI, in XAI scenarios. We begin by summarizing two visual patterns—cluster-based and regression-based patterns—that support three common XAI tasks: feature importance, feature dependency, and model accuracy. We also introduce how to select the parameters of OM4AnI based on these patterns. We construct two case studies to identify the effectiveness of OM4AnI using public datasets: Census Income dataset and MNIST dataset. OM4AnI is applied to both scenarios under various visual conditions (e.g., marker size and rendering order) to assess its effectiveness. The results demonstrate that OM4AnI serves as an effective quality metric for these two common XAI scenarios, paving the way for adapting other quality metrics to be scalable within XAI contexts.

Scatterplots are widely used across various domains to identify anomalies in datasets, particularly in multi-class settings, such as detecting misclassified or mislabeled data. However, scatterplot effectiveness often declines with large datasets due to limited display resolution. This paper introduces a novel Visual Quality Measure (VQM) - OM4AnI (Overlap Measure for Anomaly Identification) - which quantifies the degree of overlap for identifying anomalies, helping users estimate how effectively anomalies can be observed in multi-class scatterplots. OM4AnI begins by computing anomaly index based on each data point's position relative to its class cluster. The scatterplot is then discretized into a matrix representation by binning the display space into cell-level (pixel-level) grids and computing the coverage for each pixel. It takes into account the anomaly index of data points covering these pixels and visual features (marker shapes, marker sizes, and rendering orders). Building on this foundation, we sum all the coverage information in each cell (pixel) of matrix representation to obtain the final quality score with respect to anomaly identification. We conducted an evaluation to analyze the efficiency, effectiveness, sensitivity of OM4AnI in comparison with six representative baseline methods that are based on different computation granularity levels: data level, marker level, and pixel level. The results show that OM4AnI outperforms baseline methods by exhibiting more monotonic trends against the ground truth and greater sensitivity to rendering order, unlike the baseline methods. It confirms that OM4AnI can inform users about how effectively their scatterplots support anomaly identification. Overall, OM4AnI shows strong potential as an evaluation metric and for optimizing scatterplots through automatic adjustment of visual parameters.

Generating virtual organ populations that capture sufficient variability while remaining plausible is essential to conduct in silico trials (ISTs) of medical devices. However, not all anatomical shapes of interest are always available for each individual in a population. The imaging examinations and modalities used can vary between subjects depending on their individualized clinical pathways. Different imaging modalities may have various fields of view and are sensitive to signals from other tissues/organs, or both. Hence, missing/partially overlapping anatomical information is often available across individuals. We introduce a generative shape model for multipart anatomical structures, learnable from sets of unpaired datasets, i.e., where each substructure in the shape assembly comes from datasets with missing or partially overlapping substructures from disjoint subjects of the same population. The proposed generative model can synthesize complete multipart shape assemblies coined virtual chimeras (VCs). We applied this framework to build VCs from databases of whole-heart shape assemblies that each contribute samples for heart substructures. Specifically, we propose a graph neural network-based generative shape compositional framework, which comprises two components, a part-aware generative shape model that captures the variability in shape observed for each structure of interest in the training population and a spatial composition network that assembles/composes the structures synthesized by the former into multipart shape assemblies (i.e., VCs). We also propose a novel self-supervised learning scheme that enables the spatial composition network to be trained with partially overlapping data and weak labels. We trained and validated our approach using shapes of cardiac structures derived from cardiac magnetic resonance (MR) images in the UK Biobank (UKBB). When trained with complete and partially overlapping data, our approach significantly outperforms a principal component analysis (PCA)-based shape model (trained with complete data) in terms of generalizability and specificity. This demonstrates the superiority of the proposed method, as the synthesized cardiac virtual populations are more plausible and capture a greater degree of shape variability than those generated by the PCA-based shape model.

Image-to-text generation involves automatically generating descriptive text from images and has applications in medical report generation. However, traditional approaches often exhibit a semantic gap between visual and textual information. In this paper, we propose a multi-task learning framework to leverage both visual and non-imaging data for generating radiology reports. Along with chest X-ray images, 10 additional features comprising numeric, binary, categorical, and text data were incorporated to create a unified representation. The model was trained to generate text, predict the degree of patient severity, and identify medical findings. Multi-task learning, especially with text generation prioritisation, improved performance over single-task baselines across language generation metrics. The framework also mitigated overfitting in auxiliary tasks compared to single-task models. Qualitative analysis showed logically coherent narratives and accurate identification of findings, though some repetition and disjointed phrasing remained. This work demonstrates the benefits of multi-modal, multi-task learning for image-to-text generation applications.

Despite success on multi-contrast MR image synthesis, generating specific modalities remains challenging. Those include Magnetic Resonance Angiography (MRA) that highlights details of vascular anatomy using specialised imaging sequences for emphasising inflow effect. This work proposes an end-to-end generative adversarial network that can synthesise anatomically plausible, high-resolution 3D MRA images using commonly acquired multi-contrast MR images (e.g. T1/T2/PD-weighted MR images) for the same subject whilst preserving the continuity of vascular anatomy. A reliable technique for MRA synthesis would unleash the research potential of very few population databases with imaging modalities (such as MRA) that enable quantitative characterisation of whole-brain vasculature. Our work is motivated by the need to generate digital twins and virtual patients of cerebrovascular anatomy for in-silico studies and/or in-silico trials. We propose a dedicated generator and discriminator that leverage the shared and complementary features of multi-source images. We design a composite loss function for emphasising vascular properties by minimising the statistical difference between the feature representations of the target images and the synthesised outputs in both 3D volumetric and 2D projection domains. Experimental results show that the proposed method can synthesise high-quality MRA images and outperform the state-of-the-art generative models both qualitatively and quantitatively. The importance assessment reveals that T2 and PD-weighted images are better predictors of MRA images than T1; and PD-weighted images contribute to better visibility of small vessel branches towards the peripheral regions. In addition, the proposed approach can generalise to unseen data acquired at different imaging centres with different scanners, whilst synthesising MRAs and vascular geometries that maintain vessel continuity. The results show the potential for use of the proposed approach to generating digital twin cohorts of cerebrovascular anatomy at scale from structural MR images typically acquired in population imaging initiatives.

In silico tissue models (viz. numerical phantoms) provide a mechanism for evaluating quantitative models of magnetic resonance imaging. This includes the validation and sensitivity analysis of imaging biomarkers and tissue microstructure parameters. This study proposes a novel method to generate a realistic numerical phantom of myocardial microstructure. The proposed method extends previous studies by accounting for the variability of the cardiomyocyte shape, water exchange between the cardiomyocytes (intercalated discs), disorder class of myocardial microstructure, and four sheetlet orientations. In the first stage of the method, cardiomyocytes and sheetlets are generated by considering the shape variability and intercalated discs in cardiomyocyte-cardiomyocyte connections. Sheetlets are then aggregated and oriented in the directions of interest. The morphometric study demonstrates no significant difference (p>0.01) between the distribution of volume, length, and primary and secondary axes of the numerical and real (literature) cardiomyocyte data. Moreover, structural correlation analysis validates that the in-silico tissue is in the same class of disorderliness as the real tissue. Additionally, the absolute angle differences between the simulated helical angle (HA) and input HA (reference value) of the cardiomyocytes (4.3°±3.1°) demonstrate a good agreement with the absolute angle difference between the measured HA using experimental cardiac diffusion tensor imaging (cDTI) and histology (reference value) reported by (Holmes et al., 2000) (3.7°±6.4°) and (Scollan et al. 1998) (4.9°±14.6°). Furthermore, the angular distance between eigenvectors and sheetlet angles of the input and simulated cDTI is much smaller than those between measured angles using structural tensor imaging (as a gold standard) and experimental cDTI. Combined with the qualitative results, these results confirm that the proposed method can generate richer numerical phantoms for the myocardium than previous studies.

Accurate 3D modelling of cardiac chambers is essential for clinical assessment of cardiac volume and function, including structural, and motion analysis. Furthermore, to study the correlation between cardiac morphology and other patient information within a large population, it is necessary to automatically generate cardiac mesh models of each subject within the population. In this study, we introduce MCSI-Net (Multi-Cue Shape Inference Network), where we embed a statistical shape model inside a convolutional neural network and leverage both phenotypic and demographic information from the cohort to infer subject-specific reconstructions of all four cardiac chambers in 3D. In this way, we leverage the ability of the network to learn the appearance of cardiac chambers in cine cardiac magnetic resonance (CMR) images, and generate plausible 3D cardiac shapes, by constraining the prediction using a shape prior, in the form of the statistical modes of shape variation learned a priori from a subset of the population. This, in turn, enables the network to generalise to samples across the entire population. To the best of our knowledge, this is the first work that uses such an approach for patient-specific cardiac shape generation. MCSI-Net is capable of producing accurate 3D shapes using just a fraction (about 23% to 46%) of the available image data, which is of significant importance to the community as it supports the acceleration of CMR scan acquisitions. Cardiac MR images from the UK Biobank were used to train and validate the proposed method. We also present the results from analysing 40,000 subjects of the UK Biobank at 50 time-frames, totalling two million image volumes. Our model can generate more globally consistent heart shape than that of manual annotations in the presence of inter-slice motion and shows strong agreement with the reference ranges for cardiac structure and function across cardiac ventricles and atria.

Cardiac MR acquisition with complete coverage from base to apex is required to ensure accurate subsequent analyses, such as volumetric and functional measurements. However, this requirement cannot be guaranteed when acquiring images in the presence of motion induced by cardiac muscle contraction and respiration. To address this problem, we propose an effective two-stage pipeline for detecting and synthesising absent slices in both the apical and basal region. The detection model comprises several dense blocks containing convolutional long short-term memory (ConvLSTM) layers, to leverage through-plane contextual and sequential ordering information of slices in cine MR data and achieve reliable classification results. The imputation network is based on a dedicated conditional generative adversarial network (GAN) that helps retain key visual cues and fine structural details in the synthesised image slices. The proposed network can infer multiple missing slices that are anatomically plausible and lead to improved accuracy of subsequent analyses on cardiac MRIs, e.g., ventricle segmentation, cardiac quantification compared to those derived from incomplete cardiac MR datasets. For instance, the results obtained when compensating for the absence of two basal slices show that the mean differences to the reference of stroke volume and ejection fraction are only -1.3 mL and -1.0%, respectively, which are significantly smaller than those calculated from the incomplete data (-26.8 mL and -6.7%). The proposed approach can improve the reliability of high-throughput image analysis in large-scale population studies, minimising the need for re-scanning patients or discarding incomplete acquisitions.

Shape reconstruction from sparse point clouds/images is a challenging and relevant task required for a variety of applications in computer vision and medical image analysis (e.g. surgical navigation, cardiac motion analysis, augmented/virtual reality systems). A subset of such methods, viz. 3D shape reconstruction from 2D contours, is especially relevant for computer-aided diagnosis and intervention applications involving meshes derived from multiple 2D image slices, views or projections. We propose a deep learning architecture, coined Mesh Reconstruction Network (MR-Net), which tackles this problem. MR-Net enables accurate 3D mesh reconstruction in real-time despite missing data and with sparse annotations. Using 3D cardiac shape reconstruction from 2D contours defined on short-axis cardiac magnetic resonance image slices as an exemplar, we demonstrate that our approach consistently outperforms state-of-the-art techniques for shape reconstruction from unstructured point clouds. Our approach can reconstruct 3D cardiac meshes to within 2.5-mm point-to-point error, concerning the ground-truth data (the original image spatial resolution is ∼1.8×1.8×10mm3). We further evaluate the robustness of the proposed approach to incomplete data, and contours estimated using an automatic segmentation algorithm. MR-Net is generic and could reconstruct shapes of other organs, making it compelling as a tool for various applications in medical image analysis.

High-resolution (HR), isotropic cardiac Magnetic Resonance (MR) cine imaging is challenging since it requires long acquisition and patient breath-hold times. Instead, 2D balanced steady-state free precession (SSFP) sequence is widely used in clinical routine. However, it produces highly-anisotropic image stacks, with large through-plane spacing that can hinder subsequent image analysis. To resolve this, we propose a novel, robust adversarial learning super-resolution (SR) algorithm based on conditional generative adversarial nets (GANs), that incorporates a state-of-the-art optical flow component to generate an auxiliary image to guide image synthesis. The approach is designed for real-world clinical scenarios and requires neither multiple low-resolution (LR) scans with multiple views, nor the corresponding HR scans, and is trained in an end-to-end unsupervised transfer learning fashion. The designed framework effectively incorporates visual properties and relevant structures of input images and can synthesise 3D isotropic, anatomically plausible cardiac MR images, consistent with the acquired slices. Experimental results show that the proposed SR method outperforms several state-of-the-art methods both qualitatively and quantitatively. We show that subsequent image analyses including ventricle segmentation, cardiac quantification, and non-rigid registration can benefit from the super-resolved, isotropic cardiac MR images, to produce more accurate quantitative results, without increasing the acquisition time. The average Dice similarity coefficient (DSC) for the left ventricular (LV) cavity and myocardium are 0.95 and 0.81, respectively, between real and synthesised slice segmentation. For non-rigid registration and motion tracking through the cardiac cycle, the proposed method improves the average DSC from 0.75 to 0.86, compared to the original resolution images.

Quantitative assessment of cardiac left ventricle (LV) morphology is essential to assess cardiac function and improve the diagnosis of different cardiovascular diseases. In current clinical practice, LV quantification depends on the measurement of myocardial shape indices, which is usually achieved by manual contouring of the endo- and epicardial. However, this process subjected to inter and intra-observer variability, and it is a time-consuming and tedious task. In this article, we propose a spatio-temporal multi-task learning approach to obtain a complete set of measurements quantifying cardiac LV morphology, regional-wall thickness (RWT), and additionally detecting the cardiac phase cycle (systole and diastole) for a given 3D Cine-magnetic resonance (MR) image sequence. We first segment cardiac LVs using an encoder-decoder network and then introduce a multitask framework to regress 11 LV indices and classify the cardiac phase, as parallel tasks during model optimization. The proposed deep learning model is based on the 3D spatio-temporal convolutions, which extract spatial and temporal features from MR images. We demonstrate the efficacy of the proposed method using cine-MR sequences of 145 subjects and comparing the performance with other state-of-the-art quantification methods. The proposed method obtained high prediction accuracy, with an average mean absolute error (MAE) of 129 mm2, 1.23 mm, 1.76 mm, Pearson correlation coefficient (PCC) of 96.4%, 87.2%, and 97.5% for LV and myocardium (Myo) cavity regions, 6 RWTs, 3 LV dimensions, and an error rate of 9.0% for phase classification. The experimental results highlight the robustness of the proposed method, despite varying degrees of cardiac morphology, image appearance, and low contrast in the cardiac MR sequences.

Deep learning models are sensitive to domain shift phenomena. A model trained on images from one domain cannot generalise well when tested on images from a different domain, despite capturing similar anatomical structures. It is mainly because the data distribution between the two domains is different. Moreover, creating annotation for every new modality is a tedious and time-consuming task, which also suffers from high inter- and intra- observer variability. Unsupervised domain adaptation (UDA) methods intend to reduce the gap between source and target domains by leveraging source domain labelled data to generate labels for the target domain. However, current state-of-the-art (SOTA) UDA methods demonstrate degraded performance when there is insufficient data in source and target domains. In this paper, we present a novel UDA method for multi-modal cardiac image segmentation. The proposed method is based on adversarial learning and adapts network features between source and target domain in different spaces. The paper introduces an end-to-end framework that integrates: a) entropy minimization, b) output feature space alignment and c) a novel point-cloud shape adaptation based on the latent features learned by the segmentation model. We validated our method on two cardiac datasets by adapting from the annotated source domain, bSSFP-MRI (balanced Steady-State Free Procession-MRI), to the unannotated target domain, LGE-MRI (Late-gadolinium enhance-MRI), for the multi-sequence dataset; and from MRI (source) to CT (target) for the cross-modality dataset. The results highlighted that by enforcing adversarial learning in different parts of the network, the proposed method delivered promising performance, compared to other SOTA methods.

Cardiovascular diseases are the most common cause of mortality worldwide. Detection of atrial fibrillation (AF) in the asymptomatic stage can help prevent strokes. It also improves clinical decision making through the delivery of suitable treatment such as, anticoagulant therapy, in a timely manner. The clinical significance of such early detection of AF in electrocardiogram (ECG) signals has inspired numerous studies in recent years, of which many aim to solve this task by leveraging machine learning algorithms. ECG datasets containing AF samples, however, usually suffer from severe class imbalance, which if unaccounted for, affects the performance of classification algorithms. Data augmentation is a popular solution to tackle this problem. In this study, we investigate the impact of various data augmentation algorithms, e.g., oversampling, Gaussian Mixture Models (GMMs) and Generative Adversarial Networks (GANs), on solving the class imbalance problem. These algorithms are quantitatively and qualitatively evaluated, compared and discussed in detail. The results show that deep learning-based AF signal classification methods benefit more from data augmentation using GANs and GMMs, than oversampling. Furthermore, the GAN results in circa 3% better AF classification accuracy in average while performing comparably to the GMM in terms of f1-score.

A probabilistic group-wise similarity registration technique based on Student's t-mixture model (TMM) and a multi-resolution extension of the same (mr-TMM) are proposed in this study, to robustly align shapes and establish valid correspondences, for the purpose of training statistical shape models (SSMs). Shape analysis across large cohorts requires automatic generation of the requisite training sets. Automated segmentation and landmarking of medical images often result in shapes with varying proportions of outliers and consequently require a robust method of alignment and correspondence estimation. Both TMM and mrTMM are validated by comparison with state-of-the-art registration algorithms based on Gaussian mixture models (GMMs), using both synthetic and clinical data. Four clinical data sets are used for validation: (a) 2D femoral heads (K= 1000 samples generated from DXA images of healthy subjects); (b) control-hippocampi (K= 50 samples generated from T1-weighted magnetic resonance (MR) images of healthy subjects); (c) MCI-hippocampi (K= 28 samples generated from MR images of patients diagnosed with mild cognitive impairment); and (d) heart shapes comprising left and right ventricular endocardium and epicardium (K= 30 samples generated from short-axis MR images of: 10 healthy subjects, 10 patients diagnosed with pulmonary hypertension and 10 diagnosed with hypertrophic cardiomyopathy). The proposed methods significantly outperformed the state-of-the-art in terms of registration accuracy in the experiments involving synthetic data, with mrTMM offering significant improvement over TMM. With the clinical data, both methods performed comparably to the state-of-the-art for the hippocampi and heart data sets, which contained few outliers. They outperformed the state-of-the-art for the femur data set, containing large proportions of outliers, in terms of alignment accuracy, and the quality of SSMs trained, quantified in terms of generalization, compactness and specificity.

Three-dimensional reconstructions of coronary arteries can overcome some of the limitations of 2D X-ray angiography, namely artery overlap/foreshortening and lack of depth information. Model-based arterial reconstruction algorithms usually rely on 2D coronary artery segmentations and require good robustness to outliers. In this paper, we propose a novel probabilistic method to reconstruct coronary artery centrelines from retrospectively gated X-ray images based on a probabilistic mixture model. Specifically, 3D coronary artery centrelines are described by a mixture of Student’s t-distributions, and the reconstruction is formulated as maximum-likelihood estimation of the mixture model parameters, given the 2D segmentations of arteries from 2D X-ray images. Our method provides robustness against the erroneously segmented parts in the 2D segmentations by taking advantage of the inherent robustness of t-distributions. We validate our reconstruction results using synthetic phantom and clinical X-ray angiography data. The results show that the proposed method can cope with imperfect and noisy segmentation data.

A novel probabilistic, group-wise rigid registration framework is proposed in this study, to robustly align and establish correspondence across anatomical shapes represented as unstructured point sets. Student’s t-mixture model (TMM) is employed to exploit their inherent robustness to outliers. The primary application for such a framework is the automatic construction of statistical shape models (SSMs) of anatomical structures, from medical images. Tools used for automatic segmentation and landmarking of medical images often result in segmentations with varying proportions of outliers. The proposed approach is able to robustly align shapes and establish valid correspondences in the presence of considerable outliers and large variations in shape. A multi-resolution registration (mrTMM) framework is also formulated, to further improve the performance of the proposed TMM-based registration method. Comparisons with a state-of-the art approach using clinical data show that the mrTMM method in particular, achieves higher alignment accuracy and yields SSMs that generalise better to unseen shapes.