Deep learning-based medical image-to-mesh reconstruction has rapidly evolved, enabling the transformation of medical imaging data into three-dimensional mesh models that are critical in computational medicine and in silico trials for advancing our understanding of disease mechanisms, and diagnostic and therapeutic techniques in modern medicine. This survey systematically categorizes existing approaches into four main categories: template models, statistical models, generative models, and implicit models. Each category is analysed in detail, examining their methodological foundations, strengths, limitations, and applicability to different anatomical structures and imaging modalities. We provide an extensive evaluation of these methods across various anatomical applications, from cardiac imaging to neurological studies, supported by quantitative comparisons using standard metrics. Additionally, we compile and analyse major public datasets available for medical mesh reconstruction tasks and discuss commonly used evaluation metrics and loss functions. The survey identifies current challenges in the field, including requirements for topological correctness, geometric accuracy, and multi-modality integration. Finally, we present promising future research directions in this domain. This systematic review aims to serve as a comprehensive reference for researchers and practitioners in medical image analysis and computational medicine.

Deep learning-based image registration methods have shown state-of-the-art performance and rapid inference speeds. Despite these advances, many existing approaches fall short in capturing spatially varying information in non-local regions of feature maps due to the reliance on spatially-shared convolution kernels. This limitation leads to suboptimal estimation of deformation fields. In this paper, we propose a 3D Spatial-Awareness Convolution Block (SACB) to enhance the spatial information within feature representations. Our SACB estimates the spatial clusters within feature maps by leveraging feature similarity and subsequently parameterizes the adaptive convolution kernels across diverse regions. This adaptive mechanism generates the convolution kernels (weights and biases) tailored to spatial variations, thereby enabling the network to effectively capture spatially varying information. Building on SACB, we introduce a pyramid flow estimator (named SACB-Net) that integrates SACBs to facilitate multi-scale flow composition, particularly addressing large deformations. Experimental results on the brain IXI and LPBA datasets as well as Abdomen CT datasets demonstrate the effectiveness of SACB and the superiority of SACB-Net over the state-of-the-art learning-based registration methods. The code is available at https://github.com/x-xc/SACB_Net.

Intracranial aneurysms (IAs) are a critical public health concern: they are asymptomatic and can lead to fatal subarachnoid hemorrhage in case of rupture. Neuroradiologists rely on advanced imaging techniques to identify aneurysms in a patient and consider the characteristics of IAs along with several other patient-related factors for rupture risk assessment and treatment decision-making. The process of diagnostic image reading is time-intensive and prone to inter- and intra-individual variations, so researchers have proposed many computer-aided diagnosis (CAD) systems for aneurysm detection and segmentation. This paper provides a comprehensive literature survey of semi-automated and automated approaches for IA detection and segmentation and proposes a taxonomy to classify the approaches. We also discuss the current issues and give some insight into the future direction of CAD systems for IA detection and segmentation.

Intravoxel incoherent motion (IVIM) imaging is increasingly recognised as an important tool in clinical MRI, where tissue perfusion and diffusion information can aid disease diagnosis, monitoring of patient recovery, and treatment outcome assessment. Currently, the discovery of biomarkers based on IVIM imaging, similar to other medical imaging modalities, is dependent on long preclinical and clinical validation pathways to link observable markers derived from images with the underlying pathophysiological mechanisms. To speed up this process, virtual IVIM imaging is proposed. This approach provides an efficient virtual imaging tool to design, evaluate, and optimise novel approaches for IVIM imaging. In this work, virtual IVIM imaging is developed through a new finite element solver, SpinDoctor-IVIM, which extends SpinDoctor, a diffusion MRI simulation toolbox. SpinDoctor-IVIM simulates IVIM imaging signals by solving the generalised Bloch–Torrey partial differential equation. The input velocity to SpinDoctor-IVIM is computed using HemeLB, an established Lattice Boltzmann blood flow simulator. Contrary to previous approaches, SpinDoctor-IVIM accounts for volumetric microvasculature during blood flow simulations, incorporates diffusion phenomena in the intravascular space, and accounts for the permeability between the intravascular and extravascular spaces. The above-mentioned features of the proposed framework are illustrated with simulations on a realistic microvasculature model. • Presents a framework for intravoxel motion MRI with virtual imaging. • Intravascular diffusivity has a significant effect on intravoxel incoherent motion MRI signals. • Intravoxel incoherent motion signals can affect blood pressure. • Investigates the effects of vascular permeability on intravoxel incoherent motion MRI signals.

As artificial intelligence and digital medicine increasingly permeate healthcare systems, robust governance frameworks are essential to ensure ethical, secure, and effective implementation. In this context, medical image retrieval becomes a critical component of clinical data management, playing a vital role in decision-making and safeguarding patient information. Existing methods usually learn hash functions using bottleneck features, which fail to produce representative hash codes from blended embeddings. Although contrastive hashing has shown superior performance, current approaches often treat image retrieval as a classification task, using category labels to create positive/negative pairs. Moreover, many methods fail to address the out-of-distribution (OOD) issue when models encounter external OOD queries or adversarial attacks. In this work, we propose a novel method to consolidate knowledge of hierarchical features and optimization functions. We formulate the knowledge consolidation by introducing Depth-aware Representation Fusion (DaRF) and Structure-aware Contrastive Hashing (SCH). DaRF adaptively integrates shallow and deep representations into blended features, and SCH incorporates image fingerprints to enhance the adaptability of positive/negative pairings. These blended features further facilitate OOD detection and content-based recommendation, contributing to a secure AI-driven healthcare environment. Moreover, we present a content-guided ranking to improve the robustness and reproducibility of retrieval results. Our comprehensive assessments demonstrate that the proposed method could effectively recognize OOD samples and significantly outperform existing approaches in medical image retrieval (p < 0 . 05 ). In particular, our method achieves a 5.6–38.9% improvement in mean Average Precision on the anatomical radiology dataset. • Structure-aware pairing using image fingerprints to address over-centralized issues. • A novel model to consolidate hierarchical embeddings for representation learning. • Addressing ill-posed gradient issues introduced by relaxed Hamming distance. • A self-supervised OOD detection module by evaluating image reconstruction disparity. • Content-guided ranking mechanism for robust and precise retrieval.

We propose a novel recurrent variational netwo8=k]irk, SegMorph, to perform concurrent segmentation and motion estimation on cardiac cine magnetic resonance image (CMR) sequences. Our model establishes a recurrent latent space that captures spatiotemporal features from cine-MRI sequences for multitask inference and synthesis. The proposed model follows a recurrent variational auto-encoder framework and adopts a learnt prior from the temporal inputs. We utilise a multi-branch decoder to handle bi-ventricular segmentation and motion estimation simultaneously. In addition to the spatiotemporal features from the latent space, motion estimation enriches the supervision of sequential segmentation tasks by providing pseudo-ground truth. On the other hand, the segmentation branch helps with motion estimation by predicting deformation vector fields (DVFs) based on anatomical information. Experimental results demonstrate that the proposed method performs better than state-of-the-art approaches qualitatively and quantitatively for both segmentation and motion estimation tasks. We achieved an 81% average Dice Similarity Coefficient (DSC) and a less than 3.5 mm average Hausdorff distance on segmentation. Meanwhile, we achieved a motion estimation Dice Similarity Coefficient of over 79%, with approximately 0.14% of pixels displaying a negative Jacobian determinant in the estimated DVFs.

Generative modelling for shapes is a prerequisite for In-Silico Clinical Trials (ISCTs), which aim to cost-effectively validate medical device interventions using synthetic anatomical shapes, often represented as 3D surface meshes. However, constructing AI models to generate shapes closely resembling the real mesh samples is challenging due to variable vertex counts, connectivities, and the lack of dense vertex-wise correspondences across the training data. Employing graph representations for meshes, we develop a novel unsupervised geometric deep-learning model to establish refinable shape correspondences in a latent space, construct a population-derived atlas and generate realistic synthetic shapes. We additionally extend our proposed base model to a joint shape generative-clustering multi-atlas framework to incorporate further variability and preserve more details in the generated shapes. Experimental results using liver and left-ventricular models demonstrate the approach's applicability to computational medicine, highlighting its suitability for ISCTs through a comparative analysis.

Brain disorder diagnosis via resting-state functional magnetic resonance imaging (rs-fMRI) is usually limited due to the complex imaging features and sample size. For brain disorder diagnosis, the graph convolutional network (GCN) has achieved remarkable success by capturing interactions between individuals and the population. However, there are mainly three limitations: 1) The previous GCN approaches consider the non-imaging information in edge construction but ignore the sensitivity differences of features to non-imaging information. 2) The previous GCN approaches solely focus on establishing interactions between subjects (i.e., individuals and the population), disregarding the essential relationship between features. 3) Multisite data increase the sample size to help classifier training, but the inter-site heterogeneity limits the performance to some extent. This paper proposes a knowledge-aware multisite adaptive graph Transformer to address the above problems. First, we evaluate the sensitivity of features to each piece of non-imaging information, and then construct feature-sensitive and feature-insensitive subgraphs. Second, after fusing the above subgraphs, we integrate a Transformer module to capture the intrinsic relationship between features. Third, we design a domain adaptive GCN using multiple loss function terms to relieve data heterogeneity and to produce the final classification results. Last, the proposed framework is validated on two brain disorder diagnostic tasks. Experimental results show that the proposed framework can achieve state-of-the-art performance.

Generating virtual organ populations that capture sufficient variability while remaining plausible is essential to conduct in silico trials (ISTs) of medical devices. However, not all anatomical shapes of interest are always available for each individual in a population. The imaging examinations and modalities used can vary between subjects depending on their individualized clinical pathways. Different imaging modalities may have various fields of view and are sensitive to signals from other tissues/organs, or both. Hence, missing/partially overlapping anatomical information is often available across individuals. We introduce a generative shape model for multipart anatomical structures, learnable from sets of unpaired datasets, i.e., where each substructure in the shape assembly comes from datasets with missing or partially overlapping substructures from disjoint subjects of the same population. The proposed generative model can synthesize complete multipart shape assemblies coined virtual chimeras (VCs). We applied this framework to build VCs from databases of whole-heart shape assemblies that each contribute samples for heart substructures. Specifically, we propose a graph neural network-based generative shape compositional framework, which comprises two components, a part-aware generative shape model that captures the variability in shape observed for each structure of interest in the training population and a spatial composition network that assembles/composes the structures synthesized by the former into multipart shape assemblies (i.e., VCs). We also propose a novel self-supervised learning scheme that enables the spatial composition network to be trained with partially overlapping data and weak labels. We trained and validated our approach using shapes of cardiac structures derived from cardiac magnetic resonance (MR) images in the UK Biobank (UKBB). When trained with complete and partially overlapping data, our approach significantly outperforms a principal component analysis (PCA)-based shape model (trained with complete data) in terms of generalizability and specificity. This demonstrates the superiority of the proposed method, as the synthesized cardiac virtual populations are more plausible and capture a greater degree of shape variability than those generated by the PCA-based shape model.

A fast and fully automatic design of 3D cranial implants is highly desired in cranioplasty, and is key to the treatment of skull trauma. We have defined the repair of skull defects as a 3D shape completion task by proposing a two-stage diffusion model based on the representation of 3D shapes using signed distance function (SDF). Specifically, we design a diffusion model conditioned on partial shapes, we compress the 3D shape into a compact latent representation using the encoder in the vector quantized variational autoencoder (VQ-VAE) and learn the diffusion model based on this compressed discrete representation. Encoding the latent space with the autoencoder can achieve high-quality 3D cranial shape completion. In order to accurately capture local and fine-grained shape details, the training data is geometrically encoded from a compactly learned code-book. The two-stage diffusion generator with a coarse-to-fine approach possesses precise and expressive structural modeling capabilities to ensure the supplementation of detailed geometric information. Experimental results verified sufficient expressiveness of our model with generating high-fidelity results with fine-grained local details, outperforming the state-of-the-art methods.

Automatic detecting intracranial aneurysms (IAs) poses significant challenges due to their diversity, varying locations, and complex classifications by size, shape, and phenotype. Current shape-based IAs detection methods, while promising, often neglect the topological connectivity of IAs vertices and the variable traits of the aneurysm's neck, leading to fragmented detections. To address these issues, we present a mesh convolutional neural network based on gauge equivariant convolution to leverage the topological and geometric features of 3D mesh models. Our network comprises four key components: anisotropic message passing (AMP) on mesh surfaces, gauge equivariant convolution (GEC), vector-aware feature reconstruction (VFR), and a pooling-free convolutional architecture. AMP ensures accurate detection of IAs from surrounding vessels by utilizing topological connectivity and anisotropic relationships between mesh vertices. GEC offers rotational equivariance for consistently learning geometric features, improving feature learning stability and efficiency. VFR preserves the geometric and directional integrity of the vector features, enriching the representational capacity of the network. The pooling-free convolutional architecture captures local and global geometric nuances of 3D meshes, achieving precise IAs detection and producing sharper IAs boundaries. Tests on the IntrA dataset show our method outperforms the current best by 1.83% and 1.02% in mIoU and mDSC, respectively.

Automatic lung organ segmentation on computerized tomography images is crucial for lung disease diagnosis. However, the unlimited voxel values and class imbalance of lung organs can lead to false-negative/positive and leakage issues in numerous state-of-the-art methods. In addition, some lung organs are easily lost during therecycleddown/up-sample procedure, e.g., bronchioles and arterioles, which can cause severe discontinuity issue. Inspired by these, this article introduces an effective lung organ segmentation method called fuzzy attention-based border rendering feature orthogonal network, which 1) integrates an efficient transformer-like fuzzy-attention module into deep networks to cope with the uncertainty in feature representations; 2) decouples and depicts the lung organ regions as cube-trees by focusing only onrecycle-sampling border vulnerable points, rendering the severely discontinuous, false-negative/positive organ regions with two novel global-local cube-tree fusion and sparse patched feature orthogonal modules; 3) develops a multiscale self-knowledge guidance module to improve model performance and robustness. We have demonstrated the efficacy of proposed method on five challenging datasets of lung organ segmentation, i.e., airway and artery. All experimental results demonstrate that our method can achieve the favorable performance significantly.

The assessment of implant status and complications of Total Hip Replacement (THR) relies mainly on the clinical evaluation of the X-ray images to analyse the implant and the surrounding rigid structures. Current clinical practise depends on the manual identification of important landmarks to define the implant boundary and to analyse many features in arthroplasty X-ray images, which is time-consuming and could be prone to human error. Semantic segmentation based on the Convolutional Neural Network (CNN) has demonstrated successful results in many medical segmentation tasks. However, these networks cannot define explicit properties that lead to inaccurate segmentation, especially with the limited size of image datasets. Our work integrates clinical knowledge with CNN to segment the implant and detect important features simultaneously. This is instrumental in the diagnosis of complications of arthroplasty, particularly for loose implant and implant-closed bone fractures, where the location of the fracture in relation to the implant must be accurately determined. In this work, we define the points of interest using Gruen zones that represent the interface of the implant with the surrounding bone to build a Statistical Shape Model (SSM). We propose a multitask CNN that combines regression of pose and shape parameters constructed from the SSM and semantic segmentation of the implant. This integrated approach has improved the estimation of implant shape, from 74% to 80% dice score, making segmentation realistic and allowing automatic detection of Gruen zones. To train and evaluate our method, we generated a dataset of annotated hip arthroplasty X-ray images that will be made available.

Time-of-flight magnetic resonance angiography (TOF-MRA) is the least invasive and ionizing radiation-free approach for cerebrovascular imaging, but variations in imaging artifacts across different clinical centers and imaging vendors result in inter-site and inter-vendor heterogeneity, making its accurate and robust cerebrovascular segmentation challenging. Moreover, the limited availability and quality of annotated data pose further challenges for segmentation methods to generalize well to unseen datasets. In this paper, we construct the largest and most diverse TOF-MRA dataset (COSTA) from 8 individual imaging centers, with all the volumes manually annotated. Then we propose a novel network for cerebrovascular segmentation, namely CESAR, with the ability to tackle feature granularity and image style heterogeneity issues. Specifically, a coarse-to-fine architecture is implemented to refine cerebrovascular segmentation in an iterative manner. An automatic feature selection module is proposed to selectively fuse global long-range dependencies and local contextual information of cerebrovascular structures. A style self-consistency loss is then introduced to explicitly align diverse styles of TOF-MRA images to a standardized one. Extensive experimental results on the COSTA dataset demonstrate the effectiveness of our CESAR network against state-of-the-art methods. We have made 6 subsets of COSTA with the source code online available, in order to promote relevant research in the community.

Magnetic resonance imaging (MRI) scans often suffer from noise and low-resolution (LR), which affect the diagnosis and treatment results obtained for patients. LR images and noise come together with MRI, and the existing methods solve image super-resolution (SR) reconstruction and denoising tasks in a step-by-step manner, which influences the overall real distribution of the MRI data. In this paper, we present a simultaneous SR and denoising algorithm based on a stochastic normalizing flow (SNF), named the MR image SR and denoising model based on an SNF (SRDSNF). SRDSNF adds the encoded information of the input image as the conditional information to each reverse step of the stochastic normalizing flow, which realizes a consistent description of the spatial distribution between the reconstruction result and the input image. We introduce rangenull space decomposition and subsequence sampling strategies to enhance the consistency of the input and output data and increase the generation speed of the model. Simultaneous SR and denoising tasks experiment is carried out using the BrainWeb and NFBS datasets. The experimental results show that good SR and denoising results are obtained with fewer sampling steps, these results are consistent with the ground truths, and the structural similarity and peak signal-to-noise ratio of the results are also higher than those of the comparison methods. The proposed method demonstrates potential clinical promise.

Despite success on multi-contrast MR image synthesis, generating specific modalities remains challenging. Those include Magnetic Resonance Angiography (MRA) that highlights details of vascular anatomy using specialised imaging sequences for emphasising inflow effect. This work proposes an end-to-end generative adversarial network that can synthesise anatomically plausible, high-resolution 3D MRA images using commonly acquired multi-contrast MR images (e.g. T1/T2/PD-weighted MR images) for the same subject whilst preserving the continuity of vascular anatomy. A reliable technique for MRA synthesis would unleash the research potential of very few population databases with imaging modalities (such as MRA) that enable quantitative characterisation of whole-brain vasculature. Our work is motivated by the need to generate digital twins and virtual patients of cerebrovascular anatomy for in-silico studies and/or in-silico trials. We propose a dedicated generator and discriminator that leverage the shared and complementary features of multi-source images. We design a composite loss function for emphasising vascular properties by minimising the statistical difference between the feature representations of the target images and the synthesised outputs in both 3D volumetric and 2D projection domains. Experimental results show that the proposed method can synthesise high-quality MRA images and outperform the state-of-the-art generative models both qualitatively and quantitatively. The importance assessment reveals that T2 and PD-weighted images are better predictors of MRA images than T1; and PD-weighted images contribute to better visibility of small vessel branches towards the peripheral regions. In addition, the proposed approach can generalise to unseen data acquired at different imaging centres with different scanners, whilst synthesising MRAs and vascular geometries that maintain vessel continuity. The results show the potential for use of the proposed approach to generating digital twin cohorts of cerebrovascular anatomy at scale from structural MR images typically acquired in population imaging initiatives.

Deformable image registration (DIR) can be used to track cardiac motion. Conventional DIR algorithms aim to establish a dense and non-linear correspondence between independent pairs of images. They are, nevertheless, computationally intensive and do not consider temporal dependencies to regulate the estimated motion in a cardiac cycle. In this paper, leveraging deep learning methods, we formulate a novel hierarchical probabilistic model, termed DragNet, for fast and reliable spatio-temporal registration in cine cardiac magnetic resonance (CMR) images and for generating synthetic heart motion sequences. DragNet is a variational inference framework, which takes an image from the sequence in combination with the hidden states of a recurrent neural network (RNN) as inputs to an inference network per time step. As part of this framework, we condition the prior probability of the latent variables on the hidden states of the RNN utilised to capture temporal dependencies. We further condition the posterior of the motion field on a latent variable from hierarchy and features from the moving image. Subsequently, the RNN updates the hidden state variables based on the feature maps of the fixed image and the latent variables. Different from traditional methods, DragNet performs registration on unseen sequences in a forward pass, which significantly expedites the registration process. Besides, DragNet enables generating a large number of realistic synthetic image sequences given only one frame, where the corresponding deformations are also retrieved. The probabilistic framework allows for computing spatio-temporal uncertainties in the estimated motion fields. Our results show that DragNet performance is comparable with state-of-the-art methods in terms of registration accuracy, with the advantage of offering analytical pixel-wise motion uncertainty estimation across a cardiac cycle and being a motion generator. We will make our code publicly available.

The automatic and accurate analysis of medical images (e.g., segmentation,detection, classification) are prerequisites for modern disease diagnosis and prognosis. Computer-aided diagnosis (CAD) systems empower accurate and effective detection of various diseases and timely treatment decisions. The past decade witnessed a spur in deep learning (DL)-based CADs showing outstanding performance across many health care applications. Medical imaging is hindered by multiple sources of uncertainty ranging fromnteasurement (aleatoric) errors, physiological variability, and limited medical knowledge (epistemic errors). However, uncertainty quantification (UQ) in most existing DL methods is insufficiently investigated, particularly in medical image analysis. Therefore, to address this gap, in this article, we propose a simple yet novel hierarchical attentive multilevel feature fusion model with an uncertainty-aware module for medical image classification coinedHercules. This approach is tested on several real medical image classification challenges. The proposedHerculesmodel consists of two main feature fusion blocks, where the former concentrates on attention-based fusion with uncertainty quantification module and the latter uses the raw features.Herculeswas evaluated across three medical imaging datasets, i.e., retinal OCT, lung CT, and chest X-ray.Herculesproduced the best classification accuracy in retinal OCT (94.21%), lung CT (99.59%), and chest X-ray (96.50%) datasets, respectively, against other state-of-the-art medical image classification methods.

Modern radiotherapy delivers treatment plans optimised on an individual patient level, using CT-based 3D models of patient anatomy. This optimisation is fundamentally based on simple assumptions about the relationship between radiation dose delivered to the cancer (increased dose will increase cancer control) and normal tissue (increased dose will increase rate of side effects). The details of these relationships are still not well understood, especially for radiation-induced toxicity. We propose a convolutional neural network based on multiple instance learning to analyse toxicity relationships for patients receiving pelvic radiotherapy. A dataset comprising of 315 patients were included in this study; with 3D dose distributions, pre-treatment CT scans with annotated abdominal structures, and patient-reported toxicity scores provided for each participant. In addition, we propose a novel mechanism for segregating the attentions over space and dose/imaging features independently for a better understanding of the anatomical distribution of toxicity. Quantitative and qualitative experiments were performed to evaluate the network performance. The proposed network could predict toxicity with 80% accuracy. Attention analysis over space demonstrated that there was a significant association between radiation dose to the anterior and right iliac of the abdomen and patient-reported toxicity. Experimental results showed that the proposed network had outstanding performance for toxicity prediction, localisation and explanation with the ability of generalisation for an unseen dataset.

In recent years, more and more attention has been devoted to geometric deep learning (GDL) and its applications to various problems in medical imaging. Unlike convolutional neural networks (CNNs) limited to 2-D/3-D grid-structured data, GDL can handle non-Euclidean data (i.e., graphs and manifolds) and is hence well-suited for medical imaging data such as structure-function connectivity networks, imaging genetics and omics, spatio-temporal anatomical representations, physics-informed GDL for optimal imaging sampling and acquisition, GDL in imaging inverse problems, etc. However, despite recent advances in GDL research, questions remain on how best to learn representations of non-Euclidean medical imaging data; how to convolve effectively on graphs; how to perform graph pooling/unpooling; how to handle heterogeneous data; and how to improve the interpretability of GDL. After discussing many other domain experts, we identify the need for a special issue that brings to the attention of the medical imaging community these interesting topics.

For significant memory concern (SMC) and mild cognitive impairment (MCI), their classification performance is limited by confounding features, diverse imaging protocols, and limited sample size. To address the above limitations, we introduce a dual-modality fused brain connectivity network combining resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI), and propose three mechanisms in the current graph convolutional network (GCN) to improve classifier performance. First, we introduce a DTI-strength penalty term for constructing functional connectivity networks. Stronger structural connectivity and bigger structural strength diversity between groups provide a higher opportunity for retaining connectivity information. Second, a multi-center attention graph with each node representing a subject is proposed to consider the influence of data source, gender, acquisition equipment, and disease status of those training samples in GCN. The attention mechanism captures their different impacts on edge weights. Third, we propose a multi-channel mechanism to improve filter performance, assigning different filters to features based on feature statistics. Applying those nodes with low-quality features to perform convolution would also deteriorate filter performance. Therefore, we further propose a pooling mechanism, which introduces the disease status information of those training samples to evaluate the quality of nodes. Finally, we obtain the final classification results by inputting the multi-center attention graph into the multi-channel pooling GCN. The proposed method is tested on three datasets (i.e., an ADNI 2 dataset, an ADNI 3 dataset, and an in-house dataset). Experimental results indicate that the proposed method is effective and superior to other related algorithms, with a mean classification accuracy of 93.05% in our binary classification tasks. Our code is available at: https://github.com/Xuegang-S.

In silico tissue models (viz. numerical phantoms) provide a mechanism for evaluating quantitative models of magnetic resonance imaging. This includes the validation and sensitivity analysis of imaging biomarkers and tissue microstructure parameters. This study proposes a novel method to generate a realistic numerical phantom of myocardial microstructure. The proposed method extends previous studies by accounting for the variability of the cardiomyocyte shape, water exchange between the cardiomyocytes (intercalated discs), disorder class of myocardial microstructure, and four sheetlet orientations. In the first stage of the method, cardiomyocytes and sheetlets are generated by considering the shape variability and intercalated discs in cardiomyocyte-cardiomyocyte connections. Sheetlets are then aggregated and oriented in the directions of interest. The morphometric study demonstrates no significant difference (p>0.01) between the distribution of volume, length, and primary and secondary axes of the numerical and real (literature) cardiomyocyte data. Moreover, structural correlation analysis validates that the in-silico tissue is in the same class of disorderliness as the real tissue. Additionally, the absolute angle differences between the simulated helical angle (HA) and input HA (reference value) of the cardiomyocytes (4.3°±3.1°) demonstrate a good agreement with the absolute angle difference between the measured HA using experimental cardiac diffusion tensor imaging (cDTI) and histology (reference value) reported by (Holmes et al., 2000) (3.7°±6.4°) and (Scollan et al. 1998) (4.9°±14.6°). Furthermore, the angular distance between eigenvectors and sheetlet angles of the input and simulated cDTI is much smaller than those between measured angles using structural tensor imaging (as a gold standard) and experimental cDTI. Combined with the qualitative results, these results confirm that the proposed method can generate richer numerical phantoms for the myocardium than previous studies.

Accurate 3D modelling of cardiac chambers is essential for clinical assessment of cardiac volume and function, including structural, and motion analysis. Furthermore, to study the correlation between cardiac morphology and other patient information within a large population, it is necessary to automatically generate cardiac mesh models of each subject within the population. In this study, we introduce MCSI-Net (Multi-Cue Shape Inference Network), where we embed a statistical shape model inside a convolutional neural network and leverage both phenotypic and demographic information from the cohort to infer subject-specific reconstructions of all four cardiac chambers in 3D. In this way, we leverage the ability of the network to learn the appearance of cardiac chambers in cine cardiac magnetic resonance (CMR) images, and generate plausible 3D cardiac shapes, by constraining the prediction using a shape prior, in the form of the statistical modes of shape variation learned a priori from a subset of the population. This, in turn, enables the network to generalise to samples across the entire population. To the best of our knowledge, this is the first work that uses such an approach for patient-specific cardiac shape generation. MCSI-Net is capable of producing accurate 3D shapes using just a fraction (about 23% to 46%) of the available image data, which is of significant importance to the community as it supports the acceleration of CMR scan acquisitions. Cardiac MR images from the UK Biobank were used to train and validate the proposed method. We also present the results from analysing 40,000 subjects of the UK Biobank at 50 time-frames, totalling two million image volumes. Our model can generate more globally consistent heart shape than that of manual annotations in the presence of inter-slice motion and shows strong agreement with the reference ranges for cardiac structure and function across cardiac ventricles and atria.

Cardiac MR acquisition with complete coverage from base to apex is required to ensure accurate subsequent analyses, such as volumetric and functional measurements. However, this requirement cannot be guaranteed when acquiring images in the presence of motion induced by cardiac muscle contraction and respiration. To address this problem, we propose an effective two-stage pipeline for detecting and synthesising absent slices in both the apical and basal region. The detection model comprises several dense blocks containing convolutional long short-term memory (ConvLSTM) layers, to leverage through-plane contextual and sequential ordering information of slices in cine MR data and achieve reliable classification results. The imputation network is based on a dedicated conditional generative adversarial network (GAN) that helps retain key visual cues and fine structural details in the synthesised image slices. The proposed network can infer multiple missing slices that are anatomically plausible and lead to improved accuracy of subsequent analyses on cardiac MRIs, e.g., ventricle segmentation, cardiac quantification compared to those derived from incomplete cardiac MR datasets. For instance, the results obtained when compensating for the absence of two basal slices show that the mean differences to the reference of stroke volume and ejection fraction are only -1.3 mL and -1.0%, respectively, which are significantly smaller than those calculated from the incomplete data (-26.8 mL and -6.7%). The proposed approach can improve the reliability of high-throughput image analysis in large-scale population studies, minimising the need for re-scanning patients or discarding incomplete acquisitions.

The papers in this special section focus on generative adversarial networks in biomedical image computing. The field of biomedical imaging has obtained great progress from Roentgen’s original discovery of the X-ray to the current imaging tools, including Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Computed Tomography (CT), and Ultrasound (US). The benefits of using these non-invasive imaging technologies are to assess the current condition of an organ or tissue, which can be used to monitor a patient over time over time for accurate and timely diagnosis and treatment.With the development of imaging technologies, developing advanced artificial intelligence algorithms for automated image analysis has shown the potential to change many aspects of clinical applications within the next decade. Meanwhile, these advanced technologies have also brought new issues and challenges. Thus, there has been a growing demand for biomedical imaging computing to be a component of clinical trials and device improvement. Currently, Generative adversarial networks (GANs) have been attached growing interests in the computer vision community due to their capability of data generation or translation. GAN-based models are able to learn from a set of training data and generate new data with the same characteristics as the training ones, which have also proven to be the state of the art for generating sharp and realistic images. More importantly, GAN has been rapidly applied to many traditional and novel applications in the medical domain, such as image reconstruction, segmentation, diagnosis, synthesis, and so on. Despite GAN substantial progress in these areas, their application to medical image computing still faces challenges and unsolved problems remain.

Parkinson’s disease (PD) is known as an irreversible neurodegenerative disease that mainly affects the patient’s motor system. Early classification and regression of PD are essential to slow down this degenerative process from its onset. In this article, a novel adaptive unsupervised feature selection approach is proposed by exploiting manifold learning from longitudinal multimodal data. Classification and clinical score prediction are performed jointly to facilitate early PD diagnosis. Specifically, the proposed approach performs united embedding and sparse regression, which can determine the similarity matrices and discriminative features adaptively. Meanwhile, we constrain the similarity matrix among subjects and exploit the${l}_{\mathrm {2,p}}$norm to conduct sparse adaptive control for obtaining the intrinsic information of the multimodal data structure. An effective iterative optimization algorithm is proposed to solve this problem. We perform abundant experiments on the Parkinson’s Progression Markers Initiative (PPMI) data set to verify the validity of the proposed approach. The results show that our approach boosts the performance on the classification and clinical score regression of longitudinal data and surpasses the state-of-the-art approaches.

Shape reconstruction from sparse point clouds/images is a challenging and relevant task required for a variety of applications in computer vision and medical image analysis (e.g. surgical navigation, cardiac motion analysis, augmented/virtual reality systems). A subset of such methods, viz. 3D shape reconstruction from 2D contours, is especially relevant for computer-aided diagnosis and intervention applications involving meshes derived from multiple 2D image slices, views or projections. We propose a deep learning architecture, coined Mesh Reconstruction Network (MR-Net), which tackles this problem. MR-Net enables accurate 3D mesh reconstruction in real-time despite missing data and with sparse annotations. Using 3D cardiac shape reconstruction from 2D contours defined on short-axis cardiac magnetic resonance image slices as an exemplar, we demonstrate that our approach consistently outperforms state-of-the-art techniques for shape reconstruction from unstructured point clouds. Our approach can reconstruct 3D cardiac meshes to within 2.5-mm point-to-point error, concerning the ground-truth data (the original image spatial resolution is ∼1.8×1.8×10mm3). We further evaluate the robustness of the proposed approach to incomplete data, and contours estimated using an automatic segmentation algorithm. MR-Net is generic and could reconstruct shapes of other organs, making it compelling as a tool for various applications in medical image analysis.

High-resolution (HR), isotropic cardiac Magnetic Resonance (MR) cine imaging is challenging since it requires long acquisition and patient breath-hold times. Instead, 2D balanced steady-state free precession (SSFP) sequence is widely used in clinical routine. However, it produces highly-anisotropic image stacks, with large through-plane spacing that can hinder subsequent image analysis. To resolve this, we propose a novel, robust adversarial learning super-resolution (SR) algorithm based on conditional generative adversarial nets (GANs), that incorporates a state-of-the-art optical flow component to generate an auxiliary image to guide image synthesis. The approach is designed for real-world clinical scenarios and requires neither multiple low-resolution (LR) scans with multiple views, nor the corresponding HR scans, and is trained in an end-to-end unsupervised transfer learning fashion. The designed framework effectively incorporates visual properties and relevant structures of input images and can synthesise 3D isotropic, anatomically plausible cardiac MR images, consistent with the acquired slices. Experimental results show that the proposed SR method outperforms several state-of-the-art methods both qualitatively and quantitatively. We show that subsequent image analyses including ventricle segmentation, cardiac quantification, and non-rigid registration can benefit from the super-resolved, isotropic cardiac MR images, to produce more accurate quantitative results, without increasing the acquisition time. The average Dice similarity coefficient (DSC) for the left ventricular (LV) cavity and myocardium are 0.95 and 0.81, respectively, between real and synthesised slice segmentation. For non-rigid registration and motion tracking through the cardiac cycle, the proposed method improves the average DSC from 0.75 to 0.86, compared to the original resolution images.

A common feature of morphogenesis is the formation of three-dimensional structures from the folding of two-dimensional epithelial sheets, aided by cell shape changes at the cellular-level. Changes in cell shape must be studied in the context of cell-polarised biomechanical processes within the epithelial sheet. In epithelia with highly curved surfaces, finding single-cell alignment along a biological axis can be difficult to automate in silico. We present 'Origami', a MATLAB-based image analysis pipeline to compute direction-variant cell shape features along the epithelial apico-basal axis. Our automated method accurately computed direction vectors denoting the apico-basal axis in regions with opposing curvature in synthetic epithelia and fluorescence images of zebrafish embryos. As proof of concept, we identified different cell shape signatures in the developing zebrafish inner ear, where the epithelium deforms in opposite orientations to form different structures. Origami is designed to be user-friendly and is generally applicable to fluorescence images of curved epithelia.

Salivary gland ultrasonography (SGUS) has shown good potential in the diagnosis of primary Sjögren's syndrome (pSS). However, a series of international studies have reported needs for improvements of the existing pSS scoring procedures in terms of inter/intra observer reliability before being established as standardized diagnostic tools. The present study aims to solve this problem by employing radiomics features and artificial intelligence (AI) algorithms to make the pSS scoring more objective and faster compared to human expert scoring. The assessment of AI algorithms was performed on a two-centric cohort, which included 600 SGUS images (150 patients) annotated using the original SGUS scoring system proposed in 1992 for pSS. For each image, we extracted 907 histogram-based and descriptive statistics features from segmented salivary glands. Optimal feature subsets were found using the genetic algorithm based wrapper approach. Among the considered algorithms (seven classifiers and five regressors), the best preforming was the multilayer perceptron (MLP) classifier (κ = 0.7). The MLP over-performed average score achieved by the clinicians (κ = 0.67) by the considerable margin, whereas its reliability was on the level of human intra-observer variability (κ = 0.71). The presented findings indicate that the continuously increasing HarmonicSS cohort will enable further advancements in AI-based pSS scoring methods by SGUS. In turn, this may establish SGUS as an effective noninvasive pSS diagnostic tool, with the final goal to supplement current diagnostic tests.

Osteoporosis is an age-associated disease characterised by low bone mineral density (BMD) and micro-architectural deterioration leading to enhanced fracture risk. Conventional dual-energy X-ray absorptiometry (DXA) analysis has facilitated our understanding of BMD reduction in specific regions of interest (ROIs) within the femur, but cannot resolve spatial BMD patterns nor reflect age-related changes in bone microarchitecture due to its inherent averaging of pixel BMD values into large ROIs. To address these limitations and develop a comprehensive model of involutional bone loss, this paper presents a fully automatic pipeline to build a spatio-temporal atlas of ageing bone in the proximal femur. A new technique, termed DXA region free analysis (DXA RFA), is proposed to eliminate morphological variation between DXA scans by warping each image into a reference template. To construct the atlas, we use unprocessed DXA data from Caucasian women aged 20-97 years participating in three cohort studies in Western Europe ( ,000). A novel calibration procedure, termed quantile matching regression, is proposed to integrate data from different DXA manufacturers. Pixel-wise BMD evolution with ageing was modelled using smooth quantile curves. This technique enables characterisation of spatially-complex BMD change patterns with ageing, visualised using heat-maps. Furthermore, quantile curves plotted at different pixel coordinates showed consistently different rates of bone loss at different regions within the femoral neck. Given the close relationship between spatio-temporal bone loss and osteoporotic fracture, improved understanding of the bone ageing process could lead to enhanced prognostic, preventive and therapeutic strategies for the disease.

The recognition of different cell compartments, the types of cells, and their interactions is a critical aspect of quantitative cell biology. However, automating this problem has proven to be non-trivial and requires solving multi-class image segmentation tasks that are challenging owing to the high similarity of objects from different classes and irregularly shaped structures. To alleviate this, graphical models are useful due to their ability to make use of prior knowledge and model inter-class dependences. Directed acyclic graphs, such as trees, have been widely used to model top-down statistical dependences as a prior for improved image segmentation. However, using trees, a few inter-class constraints can be captured. To overcome this limitation, we propose polytree graphical models that capture label proximity relations more naturally compared to tree-based approaches. A novel recursive mechanism based on two-pass message passing was developed to efficiently calculate closed-form posteriors of graph nodes on polytrees. The algorithm is evaluated on simulated data and on two publicly available fluorescence microscopy datasets, outperforming directed trees and three state-of-the-art convolutional neural networks, namely, SegNet, DeepLab, and PSPNet. Polytrees are shown to outperform directed trees in predicting segmentation error by highlighting areas in the segmented image that do not comply with prior knowledge. This paves the way to uncertainty measures on the resulting segmentation and guide subsequent segmentation refinement.

OBJECTIVE: Automatic vascular enhancement in X-ray cineangiography is of crucial interest, for instance, for better visualizing and quantifying coronary arteries in diagnostic and interventional procedures. METHODS: A novel patch-based adaptive background subtraction method (PABSM) is proposed automatically enhancing vessels in coronary X-ray cineangiography. First, pixels in the cineangiogram are described by the vesselness and Gabor features. Second, a classifier is utilized to separate the cineangiogram into the rough vascular and non-vascular region. Dilation is applied to the classified binary image to include more vascular region. Third, a patch-based background synthesis is utilized to fill the removed vascular region. RESULTS: A database containing 320 cineangiograms of 175 patients was collected, and then an interventional cardiologist annotated all vascular structures. The performance of PABSM is compared with six state-of-the-art vascular enhancement methods regarding the precision-recall curve and C-value. The area under the precision-recall curve is 0.7133, and the C-value is 0.9659. CONCLUSION: PABSM can automatically enhance the coronary artery in the cineangiograms. It preserves the integrity of vascular topological structures, particularly in complex vascular regions, and removes noise caused by the non-uniform gray-level distribution in the cineangiogram. SIGNIFICANCE: PABSM can avoid the motion artifacts and it eases the subsequent vascular segmentation, which is crucial for the diagnosis and interventional procedures of coronary artery diseases.

Recent works show that generative adversarial networks (GANs) can be successfully applied to image synthesis and semi-supervised learning, where, given a small labeled database and a large unlabeled database, the goal is to train a powerful classifier. In this paper, we trained a retinal image synthesizer and a semi-supervised learning method for automatic glaucoma assessment using an adversarial model on a small glaucoma-labeled database and a large unlabeled database. Various studies have shown that glaucoma can be monitored by analyzing the optic disc and its surroundings, and for that reason, the images used in this paper were automatically cropped around the optic disc. The novelty of this paper is to propose a new retinal image synthesizer and a semi-supervised learning method for glaucoma assessment based on the deep convolutional GANs. In addition, and to the best of our knowledge, this system is trained on an unprecedented number of publicly available images (86926 images). This system, hence, is not only able to generate images synthetically but to provide labels automatically. Synthetic images were qualitatively evaluated using t-SNE plots of features associated with the images and their anatomical consistency was estimated by measuring the proportion of pixels corresponding to the anatomical structures around the optic disc. The resulting image synthesizer is able to generate realistic (cropped) retinal images, and subsequently, the glaucoma classifier is able to classify them into glaucomatous and normal with high accuracy (AUC = 0.9017). The obtained retinal image synthesizer and the glaucoma classifier could then be used to generate an unlimited number of cropped retinal images with glaucoma labels.

A probabilistic group-wise similarity registration technique based on Student's t-mixture model (TMM) and a multi-resolution extension of the same (mr-TMM) are proposed in this study, to robustly align shapes and establish valid correspondences, for the purpose of training statistical shape models (SSMs). Shape analysis across large cohorts requires automatic generation of the requisite training sets. Automated segmentation and landmarking of medical images often result in shapes with varying proportions of outliers and consequently require a robust method of alignment and correspondence estimation. Both TMM and mrTMM are validated by comparison with state-of-the-art registration algorithms based on Gaussian mixture models (GMMs), using both synthetic and clinical data. Four clinical data sets are used for validation: (a) 2D femoral heads (K= 1000 samples generated from DXA images of healthy subjects); (b) control-hippocampi (K= 50 samples generated from T1-weighted magnetic resonance (MR) images of healthy subjects); (c) MCI-hippocampi (K= 28 samples generated from MR images of patients diagnosed with mild cognitive impairment); and (d) heart shapes comprising left and right ventricular endocardium and epicardium (K= 30 samples generated from short-axis MR images of: 10 healthy subjects, 10 patients diagnosed with pulmonary hypertension and 10 diagnosed with hypertrophic cardiomyopathy). The proposed methods significantly outperformed the state-of-the-art in terms of registration accuracy in the experiments involving synthetic data, with mrTMM offering significant improvement over TMM. With the clinical data, both methods performed comparably to the state-of-the-art for the hippocampi and heart data sets, which contained few outliers. They outperformed the state-of-the-art for the femur data set, containing large proportions of outliers, in terms of alignment accuracy, and the quality of SSMs trained, quantified in terms of generalization, compactness and specificity.

Inferring a probability density function (pdf) for shape from a population of point sets is a challenging problem. The lack of point-to-point correspondences and the non-linearity of the shape spaces undermine the linear models. Methods based on manifolds model the shape variations naturally, however, statistics are often limited to a single geodesic mean and an arbitrary number of variation modes. We relax the manifold assumption and consider a piece-wise linear form, implementing a mixture of distinctive shape classes. The pdf for point sets is defined hierarchically, modeling a mixture of Probabilistic Principal Component Analyzers (PPCA) in higher dimension. A Variational Bayesian approach is designed for unsupervised learning of the posteriors of point set labels, local variation modes, and point correspondences. By maximizing the model evidence, the numbers of clusters, modes of variations, and points on the mean models are automatically selected. Using the predictive distribution, we project a test shape to the spaces spanned by the local PPCA's. The method is applied to point sets from: i) synthetic data, ii) healthy versus pathological heart morphologies, and iii) lumbar vertebrae. The proposed method selects models with expected numbers of clusters and variation modes, achieving lower generalization-specificity errors compared to state-of-the-art.

Statistical shape modeling is a powerful tool for visualizing and quantifying geometric and functional patterns of the heart. After myocardial infarction (MI), the left ventricle typically remodels in response to physiological challenges. Several methods have been proposed in the literature to describe statistical shape changes. Which method best characterizes left ventricular remodeling after MI is an open research question. A better descriptor of remodeling is expected to provide a more accurate evaluation of disease status in MI patients. We therefore designed a challenge to test shape characterization in MI given a set of three-dimensional left ventricular surface points. The training set comprised 100 MI patients, and 100 asymptomatic volunteers (AV). The challenge was initiated in 2015 at the Statistical Atlases and Computational Models of the Heart workshop, in conjunction with the MICCAI conference. The training set with labels was provided to participants, who were asked to submit the likelihood of MI from a different (validation) set of 200 cases (100 AV and 100 MI). Sensitivity, specificity, accuracy and area under the receiver operating characteristic curve were used as the outcome measures. The goals of this challenge were to (1) establish a common dataset for evaluating statistical shape modeling algorithms in MI, and (2) test whether statistical shape modeling provides additional information characterizing MI patients over standard clinical measures. Eleven groups with a wide variety of classification and feature extraction approaches participated in this challenge. All methods achieved excellent classification results with accuracy ranges from 0.83 to 0.98. The areas under the receiver operating characteristic curves were all above 0.90. Four methods showed significantly higher performance than standard clinical measures. The dataset and software for evaluation are available from the Cardiac Atlas Project website1.

This editorial introduces the Special Issue on Simulation and Synthesis in Medical Imaging. In this editorial, we define so-far ambiguous terms of simulation and synthesis in medical imaging. We also briefly discuss the synergistic importance of mechanistic (hypothesis-driven) and phenomenological (data-driven) models of medical image generation. Finally, we introduce the twelve papers published in this issue covering both mechanistic (5) and phenomenological (7) medical image generation. This rich selection of papers covers applications in cardiology, retinopathy, histopathology, neurosciences, and oncology. It also covers all mainstream diagnostic medical imaging modalities. We conclude the editorial with a personal view on the field and highlight some existing challenges and future research opportunities.

Multi-modality medical imaging is increasingly used for comprehensive assessment of complex diseases in either diagnostic examinations or as part of medical research trials. Different imaging modalities provide complementary information about living tissues. However, multi-modal examinations are not always possible due to adversary factors, such as patient discomfort, increased cost, prolonged scanning time, and scanner unavailability. In additionally, in large imaging studies, incomplete records are not uncommon owing to image artifacts, data corruption or data loss, which compromise the potential of multi-modal acquisitions. In this paper, we propose a weakly coupled and geometry co-regularized joint dictionary learning method to address the problem of cross-modality synthesis while considering the fact that collecting the large amounts of training data is often impractical. Our learning stage requires only a few registered multi-modality image pairs as training data. To employ both paired images and a large set of unpaired data, a cross-modality image matching criterion is proposed. Then, we propose a unified model by integrating such a criterion into the joint dictionary learning and the observed common feature space for associating cross-modality data for the purpose of synthesis. Furthermore, two regularization terms are added to construct robust sparse representations. Our experimental results demonstrate superior performance of the proposed model over state-of-the-art methods.

Magnetic Resonance Imaging (MRI) offers high-resolution in vivo imaging and rich functional and anatomical multimodality tissue contrast. In practice, however, there are challenges associated with considerations of scanning costs, patient comfort, and scanning time that constrain how much data can be acquired in clinical or research studies. In this paper, we explore the possibility of generating high-resolution and multimodal images from low-resolution single-modality imagery. We propose the weakly-supervised joint convolutional sparse coding to simultaneously solve the problems of super-resolution (SR) and cross-modality image synthesis. The learning process requires only a few registered multimodal image pairs as the training set. Additionally, the quality of the joint dictionary learning can be improved using a larger set of unpaired images. To combine unpaired data from different image resolutions/modalities, a hetero-domain image alignment term is proposed. Local image neighborhoods are naturally preserved by operating on the whole image domain (as opposed to image patches) and using joint convolutional sparse coding. The paired images are enhanced in the joint learning process with unpaired data and an additional maximum mean discrepancy term, which minimizes the dissimilarity between their feature distributions. Experiments show that the proposed method outperforms state-of-the-art techniques on both SR reconstruction and simultaneous SR and cross-modality synthesis.

Angle-closure glaucoma is a major cause of irreversible visual impairment and can be identified by measuring the anterior chamber angle (ACA) of the eye. The ACA can be viewed clearly through anterior segment optical coherence tomography (AS-OCT), but the imaging characteristics and the shapes and locations of major ocular structures can vary significantly among different AS-OCT modalities, thus complicating image analysis. To address this problem, we propose a data-driven approach for automatic AS-OCT structure segmentation, measurement, and screening. Our technique first estimates initial markers in the eye through label transfer from a hand-labeled exemplar data set, whose images are collected over different patients and AS-OCT modalities. These initial markers are then refined by using a graph-based smoothing method that is guided by AS-OCT structural information. These markers facilitate segmentation of major clinical structures, which are used to recover standard clinical parameters. These parameters can be used not only to support clinicians in making anatomical assessments, but also to serve as features for detecting anterior angle closure in automatic glaucoma screening algorithms. Experiments on Visante AS-OCT and Cirrus high-definition-OCT data sets demonstrate the effectiveness of our approach.

This paper addresses the object recognition problem using multiple-domain inputs. We present a novel approach that utilizes labeled RGB-D data in the training stage, where depth features are extracted for enhancing the discriminative capability of the original learning system that only relies on RGB images. The highly dissimilar source and target domain data are mapped into a unified feature space through transfer at both feature and classifier levels. In order to alleviate cross-domain discrepancy, we employ a state-of-the-art domain-adaptive dictionary learning algorithm that updates image representations in both domains and the classifier parameters simultaneously. The proposed method is trained on a RGB-D Object dataset and evaluated on the Caltech-256 dataset. Experimental results suggest that our approach can lead to significant performance gain over the state-of-the-art methods.

Three-dimensional reconstructions of coronary arteries can overcome some of the limitations of 2D X-ray angiography, namely artery overlap/foreshortening and lack of depth information. Model-based arterial reconstruction algorithms usually rely on 2D coronary artery segmentations and require good robustness to outliers. In this paper, we propose a novel probabilistic method to reconstruct coronary artery centrelines from retrospectively gated X-ray images based on a probabilistic mixture model. Specifically, 3D coronary artery centrelines are described by a mixture of Student’s t-distributions, and the reconstruction is formulated as maximum-likelihood estimation of the mixture model parameters, given the 2D segmentations of arteries from 2D X-ray images. Our method provides robustness against the erroneously segmented parts in the 2D segmentations by taking advantage of the inherent robustness of t-distributions. We validate our reconstruction results using synthetic phantom and clinical X-ray angiography data. The results show that the proposed method can cope with imperfect and noisy segmentation data.

A novel probabilistic, group-wise rigid registration framework is proposed in this study, to robustly align and establish correspondence across anatomical shapes represented as unstructured point sets. Student’s t-mixture model (TMM) is employed to exploit their inherent robustness to outliers. The primary application for such a framework is the automatic construction of statistical shape models (SSMs) of anatomical structures, from medical images. Tools used for automatic segmentation and landmarking of medical images often result in segmentations with varying proportions of outliers. The proposed approach is able to robustly align shapes and establish valid correspondences in the presence of considerable outliers and large variations in shape. A multi-resolution registration (mrTMM) framework is also formulated, to further improve the performance of the proposed TMM-based registration method. Comparisons with a state-of-the art approach using clinical data show that the mrTMM method in particular, achieves higher alignment accuracy and yields SSMs that generalise better to unseen shapes.

Computational fluid dynamics (CFD) is a valuable tool for studying vascular diseases, but requires long computational time. To alleviate this issue, we propose a statistical framework to predict the aneurysmal wall shear stress patterns directly from the aneurysm shape. A database of 38 complex intracranial aneurysm shapes is used to generate aneurysm morphologies and CFD simulations. The shapes and wall shear stresses are then converted to clouds of hybrid points containing both types of information. These are subsequently used to train a joint statistical model implementing a mixture of principal component analyzers. Given a new aneurysmal shape, the trained joint model is firstly collapsed to a shape only model and used to initialize the missing shear stress values. The estimated hybrid point set is further refined by projection to the joint model space. We demonstrate that our predicted patterns can achieve significant similarities to the CFD-based results. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Renal artery segmentation remained a big challenging due to its low contrast. In this paper, we present a novel graph-cut method using tensor-based distance metric for blood vessel segmentation in scale-valued images. Conventional graph-cut methods only use intensity information, which may result in failing in segmentation of small blood vessels. To overcome this drawback, this paper introduces local geometric structure information represented as tensors to find a better solution than conventional graph-cut. A Riemannian metric is utilized to calculate tensors statistics. These statistics are used in a Gaussian Mixture Model to estimate the probability distribution of the foreground and background regions. The experimental results showed that the proposed graph-cut method can segment about $$80\,\%$$ of renal arteries with 1mm precision in diameter.

Studies have demonstrated the feasibility of late Gadolinium enhancement (LGE) cardiovascular magnetic resonance (CMR) imaging for guiding the management of patients with sequelae to myocardial infarction, such as ventricular tachycardia and heart failure. Clinical implementation of these developments necessitates a reproducible and reliable segmentation of the infarcted regions. It is challenging to compare new algorithms for infarct segmentation in the left ventricle (LV) with existing algorithms. Benchmarking datasets with evaluation strategies are much needed to facilitate comparison. This manuscript presents a benchmarking evaluation framework for future algorithms that segment infarct from LGE CMR of the LV. The image database consists of 30 LGE CMR images of both humans and pigs that were acquired from two separate imaging centres. A consensus ground truth was obtained for all data using maximum likelihood estimation. Six widely-used fixed-thresholding methods and five recently developed algorithms are tested on the benchmarking framework. Results demonstrate that the algorithms have better overlap with the consensus ground truth than most of the n-SD fixed-thresholding methods, with the exception of the Full-Width-at-Half-Maximum (FWHM) fixed-thresholding method. Some of the pitfalls of fixed thresholding methods are demonstrated in this work. The benchmarking evaluation framework, which is a contribution of this work, can be used to test and benchmark future algorithms that detect and quantify infarct in LGE CMR images of the LV. The datasets, ground truth and evaluation code have been made publicly available through the website: https://www.cardiacatlas.org/web/guest/challenges.

Statistical shape models (SSMs) have been widely employed in cardiac image segmentation. However, in conditions that induce severe shape abnormality and remodeling, such as in the case of pulmonary hypertension (PH) or hypertrophic cardiomyopathy (HCM), a single SSM is rarely capable of capturing the anatomical variability in the extremes of the distribution. This work presents a new algorithm for the segmentation of severely abnormal hearts. The algorithm is highly flexible, as it does not require a priori knowledge of the involved pathology or any specific parameter tuning to be applied to the cardiac image under analysis. The fundamental idea is to approximate the gross effect of the abnormality with a virtual remodeling transformation between the patient-specific geometry and the average shape of the reference model (e.g., average normal morphology). To define this mapping, a set of landmark points are automatically identified during boundary point search, by estimating the reliability of the candidate points. With the obtained transformation, the feature points extracted from the patient image volume are then projected onto the space of the reference SSM, where the model is used to effectively constrain and guide the segmentation process. The extracted shape in the reference space is finally propagated back to the original image of the abnormal heart to obtain the final segmentation. Detailed validation with patients diagnosed with PH and HCM shows the robustness and flexibility of the technique for the segmentation of highly abnormal hearts of different pathologies.

Although modern ultrasound acquisition systems allow recording of 3D echocardiographic images, tracking anatomical structures from them is still challenging. In addition, since these images are typically created from information obtained across several cardiac cycles, it is not yet possible to acquire high-quality 3D images from patients presenting varying heart rhythms. In this paper, we propose a method to estimate the motion field from multi-plane echocardiographic images of the left ventricle, which are acquired simultaneously during a single cardiac cycle. The method integrates tri-plane B-mode and tissue Doppler images acquired at different rotation angles around the long axis of the left ventricle. It uses a diffeomorphic continuous spatio-temporal transformation model with a spherical data representation for a better interpolation in the circumferential direction. This framework allows exploiting the spatial relation among the acquired planes. In addition, higher temporal resolution of the transformation in the beam direction is achieved by uncoupling the estimation of the different components of the velocity field. The method was validated using a realistic synthetic dataset including healthy and ischemic cases, obtaining errors of 0.14 ± 0.09 mm for displacements, 0.96 ± 1.03% for longitudinal strain and 3.94 ± 4.38% for radial strain estimation. In addition, the method was also demonstrated on a healthy volunteer and two patients with ischemia.

Groupwise registration of point sets is the fundamental step in creating statistical shape models (SSMs). When the number of points on the sets varies across the population, each point set is often regarded as a spatially transformed Gaussian mixture model (GMM) sample, and the registration problem is formulated as the estimation of the underlying GMM from the training samples. Thus, each Gaussian in the mixture specifies a landmark (or model point), which is probabilistically corresponded to a training point. The Gaussian components, transformations, and probabilistic matches are often computed by an expectation-maximization (EM) algorithm. To avoid over- and under-fitting errors, the SSM should be optimized by tuning the required number of components. In this paper, rather than manually setting the number of components before training, we start from a maximal model and prune out the negligible points during the registration by a sparsity criterion. We show that by searching over the continuous space for optimal sparsity level, we can reduce the fitting errors (generalization and specificities), and thereby help the search process for a discrete number of model points. We propose an EM framework, adopting a symmetric Dirichlet distribution as a prior, to enforce sparsity on the mixture weights of Gaussians. The negligible model points are pruned by a quadratic programming technique during EM iterations. The proposed EM framework also iteratively updates the estimates of the rigid registration parameters of the point sets to the mean model. Next, we apply the principal component analysis to the registered and equal-length training point sets and construct the SSMs. This method is evaluated by learning of sparse SSMs from 15 manually segmented caudate nuclei, 24 hippocampal, and 20 prostate data sets. The generalization, specificity, and compactness of the proposed model favorably compare to a traditional EM based model.

Statistical shape models (SSM) are used to introduce shape priors in the segmentation of medical images. However, such models require large training datasets in the case of multi-object structures, since it is required to obtain not only the individual shape variations but also the relative position and orientation among objects. A solution to overcome this limitation is to model each individual shape independently. However, this approach does not take into account the relative position, orientations and shapes among the parts of an articulated object, which may result in unrealistic geometries, such as with object overlaps. In this article, we propose a new Statistical Model, the Statistical Interspace Model (SIM), which provides information about the interaction of all the individual structures by modeling the interspace between them. The SIM is described using relative position vectors between pair of points that belong to different objects that are facing each other. These vectors are divided into their magnitude and direction, each of these groups modeled as independent manifolds. The SIM was included in a segmentation framework that contains an SSM per individual object. This framework was tested using three distinct types of datasets of CT images of the spine. Results show that the SIM completely eliminated the inter-process overlap while improving the segmentation accuracy.

Spine-related disorders are amongst the most frequently encountered problems in clinical medicine. For several applications such as 1) to improve the assessment of the strength of the spine, as well as 2) to optimize the personalization of spinal interventions, image-based biomechanical modeling of the vertebrae is expected to play an important predictive role. However, this requires the construction of computational models that are subject-specific and comprehensive. In particular, they need to incorporate information about the vertebral anisotropic micro-architecture, which plays a central role in the biomechanical function of the vertebrae. In practice, however, accurate personalization of the vertebral trabeculae has proven to be difficult as its imaging in vivo is currently infeasible. Consequently, this paper presents a statistical approach for accurate prediction of the vertebral fabric tensors based on a training sample of ex vivo micro-CT images. To the best of our knowledge, this is the first predictive model proposed and validated for vertebral datasets. The method combines features selection and partial least squares regression in order to derive optimal latent variables for the prediction of the fabric tensors based on the more easily extracted shape and density information. Detailed validation with 20 ex vivo T12 vertebrae demonstrates the accuracy and consistency of the approach for the personalization of trabecular anisotropy.

Detailed segmentation of the vertebrae is an important pre-requisite in various applications of image-based spine assessment, surgery and biomechanical modeling. In particular, accurate segmentation of the processes is required for image-guided interventions, for example for optimal placement of bone grafts between the transverse processes. Furthermore, the geometry of the processes is now required in musculoskeletal models due to their interaction with the muscles and ligaments. In this paper, we present a new method for detailed segmentation of both the vertebral bodies and processes based on statistical shape decomposition and conditional models. The proposed technique is specifically developed with the aim to handle the complex geometry of the processes and the large variability between individuals. The key technical novelty in this work is the introduction of a part-based statistical decomposition of the vertebrae, such that the complexity of the subparts is effectively reduced, and model specificity is increased. Subsequently, in order to maintain the statistical and anatomic coherence of the ensemble, conditional models are used to model the statistical inter-relationships between the different subparts. For shape reconstruction and segmentation, a robust model fitting procedure is used to exclude improbable inter-part relationships in the estimation of the shape parameters. Segmentation results based on a dataset of 30 healthy CT scans and a dataset of 10 pathological scans show a point-to-surface error improvement of 20% and 17% respectively, and the potential of the proposed technique for detailed vertebral modeling.

Zebra fish, as a popular experimental model organism, has been frequently used in biomedical research. For observing, analysing and recording labelled transparent features in zebra fish images, it is often efficient and convenient to adopt the fluorescence microscopy. However, the acquired z-stack images are always blurred, which makes deblurring/deconvolution critical for further image analysis. In this paper, we propose a Bayesian Maximum a-Posteriori (MAP) method with the sparse image priors to solve three-dimensional (3D) deconvolution problem for Wide Field (WF) fluorescence microscopy images from zebra fish embryos. The novel sparse image priors include a global Hyper-Laplacian model and a local smooth region mask. These two kinds of prior are deployed for preserving sharp edges and suppressing ringing artifacts, respectively. Both synthetic and real WF fluorescent zebra fish embryo data are used for evaluation. Experimental results demonstrate the potential applicability of the proposed method for 3D fluorescence microscopy images, compared with state-of-the-art 3D deconvolution algorithms.

This paper presents a predictive framework for the statistical personalization of ventricular fibers. To this end, the relationship between subject-specific geometry of the left (LV) and right ventricles (RV) and fiber orientation is learned statistically from a training sample of ex vivo diffusion tensor imaging datasets. More specifically, the axes in the shape space which correlate most with the myocardial fiber orientations are extracted and used for prediction in new subjects. With this approach and unlike existing fiber models, inter-subject variability is taken into account to generate latent shape predictors that are statistically optimal to estimate fiber orientation at each individual myocardial location. The proposed predictive model was applied to the task of personalizing fibers in 10 canine subjects. The results indicate that the ventricular shapes are good predictors of fiber orientation, with an improvement of 11.4% in accuracy over the average fiber model.

We propose a technique for myocardial motion estimation based on image registration using both B-mode echocardiographic images and tissue Doppler sequences acquired interleaved. The velocity field is modeled continuously using B-splines and the spatiotemporal transform is constrained to be diffeomorphic. Images before scan conversion are used to improve the accuracy of the estimation. The similarity measure includes a model of the speckle pattern distribution of B-mode images. It also penalizes the disagreement between tissue Doppler velocities and the estimated velocity field. Registration accuracy is evaluated and compared to other alternatives using a realistic synthetic dataset, obtaining mean displacement errors of about 1 mm. Finally, the method is demonstrated on data acquired from six volunteers, both at rest and during exercise. Robustness is tested against low image quality and fast heart rates during exercise. Results show that our method provides a robust motion estimate in these situations.

Anatomical labeling of the cerebral arteries forming the Circle of Willis (CoW) enables inter-subject comparison, which is required for geometric characterization and discovering risk factors associated with cerebrovascular pathologies. We present a method for automated anatomical labeling of the CoW by detecting its main bifurcations. The CoW is modeled as rooted attributed relational graph, with bifurcations as its vertices, whose attributes are characterized as points on a Riemannian manifold. The method is first trained on a set of pre-labeled examples, where it learns the variability of local bifurcation features as well as the variability in the topology. Then, the labeling of the target vasculature is obtained as maximum a posteriori probability (MAP) estimate where the likelihood of labeling individual bifurcations is regularized by the prior structural knowledge of the graph they span. The method was evaluated by cross-validation on 50 subjects, imaged with magnetic resonance angiography, and showed a mean detection accuracy of 95%. In addition, besides providing the MAP, the method can rank the labelings. The proposed method naturally handles anatomical structural variability and is demonstrated to be suitable for labeling arterial segments of the CoW.

The 12 papers in this special issue focus on medical imaging and image computing in computational physiology. Most of the papers are focused on various aspects of the cardiovascular system across various physiological processes and observational scales.

Atlases and statistical models play important roles in the personalization and simulation of cardiac physiology. For the study of the heart, however, the construction of comprehensive atlases and spatio-temporal models is faced with a number of challenges, in particular the need to handle large and highly variable image datasets, the multi-region nature of the heart, and the presence of complex as well as small cardiovascular structures. In this paper, we present a detailed atlas and spatio-temporal statistical model of the human heart based on a large population of 3D+time multi-slice computed tomography sequences, and the framework for its construction. It uses spatial normalization based on nonrigid image registration to synthesize a population mean image and establish the spatial relationships between the mean and the subjects in the population. Temporal image registration is then applied to resolve each subject-specific cardiac motion and the resulting transformations are used to warp a surface mesh representation of the atlas to fit the images of the remaining cardiac phases in each subject. Subsequently, we demonstrate the construction of a spatio-temporal statistical model of shape such that the inter-subject and dynamic sources of variation are suitably separated. The framework is applied to a 3D+time data set of 138 subjects. The data is drawn from a variety of pathologies, which benefits its generalization to new subjects and physiological studies. The obtained level of detail and the extendability of the atlas present an advantage over most cardiac models published previously.

Computational algorithms modeling the insertion of endovascular devices, such as coil or stents, have gained an increasing interest in recent years. This scientific enthusiasm is due to the potential impact that these techniques have to support clinicians by understanding the intravascular hemodynamics and predicting treatment outcomes. In this work, a virtual coiling technique for treating image-based aneurysm models is proposed. A dynamic path planning was used to mimic the structure and distribution of coils inside aneurysm cavities, and to reach high packing densities, which is desirable by clinicians when treating with coils. Several tests were done to evaluate the performance on idealized and image-based aneurysm models. The proposed technique was validated using clinical information of real coiled aneurysms. The virtual coiling technique reproduces the macroscopic behavior of inserted coils and properly captures the densities, shapes and coil distributions inside aneurysm cavities. A practical application was performed by assessing the local hemodynamic after coiling using computational fluid dynamics (CFD). Wall shear stress and intra-aneurysmal velocities were reduced after coiling. Additionally, CFD simulations show that coils decrease the amount of contrast entering the aneurysm and increase its residence time.

The development of biophysical models of the heart has the potential to get insights in the patho-physiology of the heart, which requires to accurately modeling anatomy and function. The electrical activation sequence of the ventricles depends strongly on the cardiac conduction system (CCS). Its morphology and function cannot be observed in vivo, and therefore data available come from histological studies. We present a review on data available of the peripheral CCS including new experiments. In order to build a realistic model of the CCS we designed a procedure to extract morphological characteristics of the CCS from stained calf tissue samples. A CCS model personalized with our measurements has been built using L-systems. The effect of key unknown parameters of the model in the electrical activation of the left ventricle has been analyzed. The CCS models generated share the main characteristics of observed stained Purkinje networks. The timing of the simulated electrical activation sequences were in the physiological range for CCS models that included enough density of PMJs. These results show that this approach is a potential methodology for collecting knowledge-domain data and build improved CCS models of the heart automatically.

Previous studies have shown that the accuracy of Face Recognition Systems (FRSs) decreases with the time elapsed between enrollment and testing. The main reason for the decrease is the changes in appearance of the user due to factors such as ageing, beard growth, sun-tan etc. Self-update procedure, where the system learns the biometric characteristics of the user every time he/she interacts with it, can be used to automatically update the system. However, a commonly acknowledged problem is the corruption of biometric traits due to misclassification. In this article, we test FRS, based on three classification algorithms, on two challenging databases, GEFA and YT, with 14 279 and 31 951 images, respectively. Our results suggest that complex, state-of-the-art classifiers that make use of user-specific models, need not be the best choice for use in self updating systems. In other words, tolerance to corrupted training data decreases as the complexity of the classifier increases.

This paper introduces and evaluates a fast exact algorithm and a series of faster approximate algorithms for the computation of 3D geometric moments from an unstructured surface mesh of triangles. Being based on the object surface reduces the computational complexity of these algorithms with respect to volumetric grid-based algorithms. In contrast, it can only be applied for the computation of geometric moments of homogeneous objects. This advantage and restriction is shared with other proposed algorithms based on the object boundary. The proposed exact algorithm reduces the computational complexity for computing geometric moments up to order N with respect to previously proposed exact algorithms, from N(9) to N(6). The approximate series algorithm appears as a power series on the rate between triangle size and object size, which can be truncated at any desired degree. The higher the number and quality of the triangles, the better the approximation. This approximate algorithm reduces the computational complexity to N(3). In addition, the paper introduces a fast algorithm for the computation of 3D Zernike moments from the computed geometric moments, with a computational complexity N(4), while the previously proposed algorithm is of order N(6). The error introduced by the proposed approximate algorithms is evaluated in different shapes and the cost-benefit ratio in terms of error, and computational time is analyzed for different moment orders.

A new automatic approach for saccular intracranial aneurysm isolation is proposed in this work. Due to the inter- and intra-observer variability in manual delineation of the aneurysm neck, a definition based on a minimum cost path around the aneurysm sac is proposed that copes with this variability and is able to make consistent measurements along different data sets, as well as to automate and speedup the analysis of cerebral aneurysms. The method is based on the computation of a minimal path along a scalar field obtained on the vessel surface, to find the aneurysm neck in a robust and fast manner. The computation of the scalar field on the surface is obtained using a fast marching approach with a speed function based on the exponential of the distance from the centerline bifurcation between the aneurysm dome and the parent vessels. In order to assure a correct topology of the aneurysm sac, the neck computation is constrained to a region defined by a surface Voronoi diagram obtained from the branches of the vessel centerline. We validate this method comparing our results in 26 real cases with manual aneurysm isolation obtained using a cut-plane, and also with results obtained using manual delineations from three different observers by comparing typical morphological measures.

The accurate diagnosis of osteoporosis has gained increasing importance due to the aging of our society. Areal bone mineral density (BMD) measured by dual-energy X-ray absorptiometry (DXA) is an established criterion in the diagnosis of osteoporosis. This measure, however, is limited by its two-dimensionality. This work presents a method to reconstruct both the 3D bone shape and 3D BMD distribution of the proximal femur from a single DXA image used in clinical routine. A statistical model of the combined shape and BMD distribution is presented, together with a method for its construction from a set of quantitative computed tomography (QCT) scans. A reconstruction is acquired in an intensity based 3D-2D registration process whereby an instance of the model is found that maximizes the similarity between its projection and the DXA image. Reconstruction experiments were performed on the DXA images of 30 subjects, with a model constructed from a database of QCT scans of 85 subjects. The accuracy was evaluated by comparing the reconstructions with the same subject QCT scans. The method presented here can potentially improve the diagnosis of osteoporosis and fracture risk assessment from the low radiation dose and low cost DXA devices currently used in clinical routine.

Modern organizations need to exploit the information stored in heterogeneous and interrelated data sources, but often have no means to integrate them in a principled fashion. This general database research challenge is particularly relevant in distributed e-Science. Specifically, biomed-ical research generates a vast amount of heterogeneous data, which exceeds the current technological capacity to exploit it efficiently. Typically, service-oriented architectures are used in this context to define a unified view over all sources to be integrated. This unified schema needs to be mapped onto the underlying data sources, often including also semantic annotations. This approach suffers from high complexity and setup costs. In this paper we propose a novel application of semantic and mediation technologies, which leads to an incremental and on-demand definition of data mediation services. The so-called archetypes provide the context and semantics needed to setup such services, which significantly simplify their definition.

In this work, we propose a fast approach to compute centerlines from 3D tubular domains. This technique is based on the distance transform (DT) from the object boundaries, whose implementation together with an efficient technique to detect the points where the DT collapse, allow to obtain an initial centerline, very close to the final solution. This initial centerline and the DT computed, are used in a second step to obtain a connected, one voxel thick centerline, with a fast marching approach. The method proposed here is accurate, computationally efficient, fully automatic, is able to account for loops, and also obtains the bifurcation and end points automatically.

To build a model of the myocardium, a first step is the spatial normalization of data to a reference framework. Diffusion Tensor Imaging (DTI) provides information about tissue orientation, that is, fiber structures. Therefore, proper registration algorithms must be defined to deal with this image modality. In this paper, we propose a registration framework for cardiac DTI that takes into account the special features of DTI when applied to visualization of the myocardium. We propose a similarity measure adapted to the tensorial nature of the images, as well as an appropriate framework for averaging the DTI data sets. Results shown the advantages of the proposed methodology over other related approaches.

A new multimodal biometric database designed and acquired within the framework of the European BioSecure Network of Excellence is presented. It is comprised of more than 600 individuals acquired simultaneously in three scenarios: 1) over the Internet, 2) in an office environment with desktop PC, and 3) in indoor/outdoor environments with mobile portable hardware. The three scenarios include a common part of audio/video data. Also, signature and fingerprint data have been acquired both with desktop PC and mobile portable hardware. Additionally, hand and iris data were acquired in the second scenario using desktop PC. Acquisition has been conducted by 11 European institutions. Additional features of the BioSecure Multimodal Database (BMDB) are: two acquisition sessions, several sensors in certain modalities, balanced gender and age distributions, multimodal realistic scenarios with simple and quick tasks per modality, cross-European diversity, availability of demographic data, and compatibility with other multimodal databases. The novel acquisition conditions of the BMDB allow us to perform new challenging research and evaluation of either monomodal or multimodal biometric systems, as in the recent BioSecure Multimodal Evaluation campaign. A description of this campaign including baseline results of individual modalities from the new database is also given. The database is expected to be available for research purposes through the BioSecure Association during 2008.

The increasing volume of data describing human disease processes and the growing complexity of understanding, managing, and sharing such data presents a huge challenge for clinicians and medical researchers. This paper presents the @neurIST system, which provides an infrastructure for biomedical research while aiding clinical care, by bringing together heterogeneous data and complex processing and computing services. Although @neurIST targets the investigation and treatment of cerebral aneurysms, the system's architecture is generic enough that it could be adapted to the treatment of other diseases. Innovations in @neurIST include confining the patient data pertaining to aneurysms inside a single environment that offers clinicians the tools to analyze and interpret patient data and make use of knowledge-based guidance in planning their treatment. Medical researchers gain access to a critical mass of aneurysm related data due to the system's ability to federate distributed information sources. A semantically mediated grid infrastructure ensures that both clinicians and researchers are able to seamlessly access and work on data that is distributed across multiple sites in a secure way in addition to providing computing resources on demand for performing computationally intensive simulations for treatment planning and research.

In this paper, a statistical shape analysis method for myocardial contraction is presented that was built to detect and locate regional wall motion abnormalities (RWMA). For each slice level (base, middle, and apex), 44 short-axis magnetic resonance images were selected from healthy volunteers to train a statistical model of normal myocardial contraction using independent component analysis (ICA). A classification algorithm was constructed from the ICA components to automatically detect and localize abnormally contracting regions of the myocardium. The algorithm was validated on 45 patients suffering from ischemic heart disease. Two validations were performed; one with visual wall motion scores (VWMS) and the other with wall thickening (WT) used as references. Accuracy of the ICA-based method on each slice level was 69.93% (base), 89.63% (middle), and 72.78% (apex) when WT was used as reference, and 63.70% (base), 67.41% (middle), and 66.67% (apex) when VWMS was used as reference. From this we conclude that the proposed method is a promising diagnostic support tool to assist clinicians in reducing the subjectivity in VWMS.

This paper presents a technique to estimate and model patient-specific pulsatility of cerebral aneurysms over one cardiac cycle, using 3D rotational X-ray angiography (3DRA) acquisitions. Aneurysm pulsation is modeled as a time varying B-spline tensor field representing the deformation applied to a reference volume image, thus producing the instantaneous morphology at each time point in the cardiac cycle. The estimated deformation is obtained by matching multiple simulated projections of the deforming volume to their corresponding original projections. A weighting scheme is introduced to account for the relevance of each original projection for the selected time point. The wide coverage of the projections, together with the weighting scheme, ensures motion consistency in all directions. The technique has been tested on digital and physical phantoms that are realistic and clinically relevant in terms of geometry, pulsation and imaging conditions. Results from digital phantom experiments demonstrate that the proposed technique is able to recover subvoxel pulsation with an error lower than 10% of the maximum pulsation in most cases. The experiments with the physical phantom allowed demonstrating the feasibility of pulsation estimation as well as identifying different pulsation regions under clinical conditions.

This paper presents the system architecture of the @neurIST project, which aims at supporting the research and treatment of cerebral aneurysms by bringing together heterogeneous data, computing and complex processing services. The architecture is generic enough to adapt it to the treatment of other diseases beyond cerebral aneurysms. The paper describes the generic requirements of the system and presents the architecture, applications and middleware technologies used to realise the system and highlights the innovations in @neurIST.

One of the drawbacks of statistical shape models is their occasional failure to converge. Although visually this fact is usually easy to recognize, there is no automatic way to detect it. In this paper, we introduce a generic reliability measure for statistical shape models. It is based on a probabilistic framework and uses information extracted by the model itself during the matching process. The proposed method was validated with two variants of Active Shape Models in the context facial image analysis. Experimental results on more than 3700 facial images showed a high degree of correlation between the segmentation accuracy and the estimated reliability metric.

Active shape models bear a great promise for model-based medical image analysis. Their practical use, though, is undermined due to the need to train such models on large image databases. Automatic building of point distribution models (PDMs) has been successfully addressed and a number of autolandmarking techniques are currently available. However, the need for strategies to automatically build intensity models around each landmark has been largely overlooked in the literature. This work demonstrates the potential of creating intensity models automatically by simulating image generation. We show that it is possible to reuse a 3D PDM built from computed tomography (CT) to segment gated single photon emission computed tomography (gSPECT) studies. Training is performed on a realistic virtual population where image acquisition and formation have been modeled using the SIMIND Monte Carlo simulator and ASPIRE image reconstruction software, respectively. The dataset comprised 208 digital phantoms (4D-NCAT) and 20 clinical studies. The evaluation is accomplished by comparing point-to-surface and volume errors against a proper gold standard. Results show that gSPECT studies can be successfully segmented by models trained under this scheme with subvoxel accuracy. The accuracy in estimated LV function parameters, such as end diastolic volume, end systolic volume, and ejection fraction, ranged from 90.0% to 94.5% for the virtual population and from 87.0% to 89.5% for the clinical population.

In this work we introduce the usage of bilinear models as a means of factorising the shape variation induced by subject variability and the contraction of the human heart. We show that it is feasible to reconstruct the shape of the heart at a certain point in the cardiac cycle if we are given a small number of shapes representing the same heart at different points in the same cycle, using the bilinear model. Depending on pathology and the ratios between healthy and pathological hearts in the training set, RMS reconstruction errors measured between 1.39 and 16.58 millimetres, with a median of 6.79 and 90th percentile of 9.95 millimetres.

This work is framed in the field of statistical face analysis. In particular, the problem of accurate segmentation of prominent features of the face in frontal view images is addressed. We propose a method that generalizes linear Active Shape Models (ASMs), which have already been used for this task. The technique is built upon the development of a nonlinear intensity model, incorporating a reduced set of differential invariant features as local image descriptors. These features are invariant to rigid transformations, and a subset of them is chosen by Sequential Feature Selection for each landmark and resolution level. The new approach overcomes the unimodality and Gaussianity assumptions of classical ASMs regarding the distribution of the intensity values across the training set. Our methodology has demonstrated a significant improvement in segmentation precision as compared to the linear ASM and Optimal Features ASM (a nonlinear extension of the pioneer algorithm) in the tests performed on AR, XM2VTS, and EQUINOX databases.

Rupture of intracranial saccular aneurysms is the most common cause of spontaneous subarachnoid hemorrhage, which has significant morbidity and mortality. Although there is still controversy regarding the decision on which unruptured aneurysms should be treated, this is based primarily on their size. Nonetheless, many large lesions do not rupture whereas some small ones do. It is commonly accepted that hemodynamical factors are important to better understand the natural history of cerebral aneurysms. However, it might not always be practical to carry out a detailed computational analysis of such factors if a prompt assessment is required. Since shape is likely to be dependent on the balance between hemodynamic forces and the aneurysmal surrounding environment, an appropriate morphological 3-D characterization is likely to provide a practical surrogate to quickly evaluate the risk of rupture. In this paper, an efficient and novel methodology for 3-D shape characterization of cerebral aneurysms is described. The aneurysms are isolated by taking into account a portion of their adjacent vessels. Two methods to characterize the morphology of the aneurysms models using moment invariants have been considered: geometrical moment invariants (GMI) and Zernike moment invariants (ZMI). The results have been validated in a database containing 53 patients with a total of 31 ruptured aneurysms and 24 unruptured aneurysms. It has been found that ZMI indices are more robust than GMI, and seem to provide a reliable way to discriminate between ruptured and unruptured aneurysms. Correct rupture prediction rates of approximately equal to 80% were achieved in contrast to 66% that is found when the aspect ratio index is considered.

This paper presents a framework for weighted fusion of several Active Shape and Active Appearance Models. The approach is based on the eigenspace fusion method proposed by Hall et al., which has been extended to fuse more than two weighted eigenspaces using unbiased mean and covariance matrix estimates. To evaluate the performance of fusion, a comparative assessment on segmentation precision as well as facial verification tests are performed using the AR, EQUINOX, and XM2VTS databases. Based on the results, it is concluded that the fusion is useful when the model needs to be updated online or when the original observations are absent.

In this article a complete audio-visual speech recognition system suitable for embedded devices is presented. As visual feature extraction algorithms active shape models (ASM) and discrete cosine transformation (DCT) have been investigated and discussed for an embedded implementation. The audio-visual information integration has also been designed by taking into account device limitations. It is well known that the use of visual cues improves the recognition results especially in scenarios with high level of acoustical noise. We wanted to compare the performance of lip reading and the conventional noise reduction systems in these degraded scenarios, as well as the combination of both kinds of solutions. Important improvements are obtained especially for nonstationary background noise like voice interference, car acceleration or indicator clicks. For this kind of noise lip reading outperforms the results obtained with conventional noise reduction technologies.

This paper examines the theory of kernel Fisher discriminant analysis (KFD) in a Hilbert space and develops a two-phase KFD framework, i.e., kernel principal component analysis (KPCA) plus Fisher linear discriminant analysis (LDA). This framework provides novel insights into the nature of KFD. Based on this framework, the authors propose a complete kernel Fisher discriminant analysis (CKFD) algorithm. CKFD can be used to carry out discriminant analysis in "double discriminant subspaces." The fact that, it can make full use of two kinds of discriminant information, regular and irregular, makes CKFD a more powerful discriminator. The proposed algorithm was tested and evaluated using the FERET face database and the CENPARMI handwritten numeral database. The experimental results show that CKFD outperforms other KFD algorithms.

Hemodynamic factors are thought to be implicated in the progression and rupture of intracranial aneurysms. Current efforts aim to study the possible associations of hemodynamic characteristics such as complexity and stability of intra-aneurysmal flow patterns, size and location of the region of flow impingement with the clinical history of aneurysmal rupture. However, there are no reliable methods for measuring blood flow patterns in vivo. In this paper, an efficient methodology for patient-specific modeling and characterization of the hemodynamics in cerebral aneurysms from medical images is described. A sensitivity analysis of the hemodynamic characteristics with respect to variations of several variables over the expected physiologic range of conditions is also presented. This sensitivity analysis shows that although changes in the velocity fields can be observed, the characterization of the intra-aneurysmal flow patterns is not altered when the mean input flow, the flow division, the viscosity model, or mesh resolution are changed. It was also found that the variable that has the greater impact on the computed flow fields is the geometry of the vascular structures. We conclude that with the proposed modeling pipeline clinical studies involving large numbers cerebral aneurysms are feasible.

One of the best challenges in Lip Reading is to apply this technology in embedded devices. In current solutions the high use of resources, especially in reference to visual processing, makes the implementation and integration into a small device very difficult. In this article a new and efficient algorithm for detection and tracking of lips is presented. Lip Finding and Tracking are customary first steps in visual processing for Lip Reading. In our approach Lips are found among a small number of blobs, which should fulfill geometric constraints. The proposed algorithm runs on an ARM920T embedded device using on average less than 4 MHz1 (2,7 % of CPU load). This algorithm shows promising results in a realistic environment accomplishing successful lip finding and tracking in 94.2 % of more than 4900 image frames. 1.

In this paper, a novel image projection analysis method (UIPDA) is first developed for image feature extraction. In contrast to Liu's projection discriminant method, UIPDA has the desirable property that the projected feature vectors are mutually uncorrelated. Also, a new LDA technique called EULDA is presented for further feature extraction. The proposed methods are tested on the ORL and the NUST603 face databases. The experimental results demonstrate that: (i) UIPDA is superior to Liu's projection discriminant method and more efficient than Eigenfaces and Fisherfaces; (ii) EULDA outperforms the existing PCA plus LDA strategy; (iii) UIPDA plus EULDA is a very effective two-stage strategy for image feature extraction.

Flow-mediated dilation (FMD) offers a mechanism to characterize endothelial function and, therefore, may play a role in the diagnosis of cardiovascular diseases. Computerized analysis techniques are very desirable to give accuracy and objectivity to the measurements. Virtually all methods proposed up to now to measure FMD rely on accurate edge detection of the arterial wall, and they are not always robust in the presence of poor image quality or image artifacts. A novel method for automatic dilation assessment based on a global image analysis strategy is presented. We model interframe arterial dilation as a superposition of a rigid motion and a scaling factor perpendicular to the artery. Rigid motion can be interpreted as a global compensation for patient and probe movements, an aspect that has not been sufficiently studied before. The scaling factor explains arterial dilation. The ultrasound sequence is analyzed in two phases using image registration to recover both transformation models. Temporal continuity in the registration parameters along the sequence is enforced with a Kalman filter since the dilation process is known to be a gradual physiological phenomenon. Comparing automated and gold standard measurements (average of manual measurements) we found a negligible bias (0.05%FMD) and a small standard deviation (SD) of the differences (1.05%FMD). These values are comparable with those obtained from manual measurements (bias = 0.23%FMD, SD(intra-obs) = 1.13%FMD, SD(inter-obs) 1.20%FMD). The proposed method offers also better reproducibility (CV = 0.40%) than the manual measurements (CV = 1.04%).

In this paper, we show how the concept of statistical deformation models (SDMs) can be used for the construction of average models of the anatomy and their variability. SDMs are built by performing a statistical analysis of the deformations required to map anatomical features in one subject into the corresponding features in another subject. The concept of SDMs is similar to statistical shape models (SSMs) which capture statistical information about shapes across a population, but offers several advantages over SSMs. First, SDMs can be constructed directly from images such as three-dimensional (3-D) magnetic resonance (MR) or computer tomography volumes without the need for segmentation which is usually a prerequisite for the construction of SSMs. Instead, a nonrigid registration algorithm based on free-form deformations and normalized mutual information is used to compute the deformations required to establish dense correspondences between the reference subject and the subjects in the population class under investigation. Second, SDMs allow the construction of an atlas of the average anatomy as well as its variability across a population of subjects. Finally, SDMs take the 3-D nature of the underlying anatomy into account by analysing dense 3-D deformation fields rather than only information about the surface shape of anatomical structures. We show results for the construction of anatomical models of the brain from the MR images of 25 different subjects. The correspondences obtained by the nonrigid registration are evaluated using anatomical landmark locations and show an average error of 1.40 mm at these anatomical landmark positions. We also demonstrate that SDMs can be constructed so as to minimize the bias toward the chosen reference subject.

A framework to analyze the propagation of measurement noise through backprojection reconstruction algorithms in electrical impedance tomography (EIT) is presented. Two measurement noise sources were considered: noise in the current drivers and in the voltage detectors. The influence of the acquisition system architecture (serial/semi-parallel) is also discussed. Three variants of backprojection reconstruction are studied: basic (unweighted), weighted and exponential backprojection. The results of error propagation theory have been compared with those obtained from simulated and experimental data. This comparison shows that the approach provides a good estimate of the reconstruction error variance. It is argued that the reconstruction error in EIT images obtained via backprojection can be approximately modeled as a spatially nonstationary Gaussian distribution. This methodology allows us to develop a spatial characterization of the reconstruction error in EIT images.

A novel method is introduced for the generation of landmarks for three-dimensional (3-D) shapes and the construction of the corresponding 3-D statistical shape models. Automatic landmarking of a set of manual segmentations from a class of shapes is achieved by 1) construction of an atlas of the class, 2) automatic extraction of the landmarks from the atlas, and 3) subsequent propagation of these landmarks to each example shape via a volumetric nonrigid registration technique using multiresolution B-spline deformations. This approach presents some advantages over previously published methods: it can treat multiple-part structures and requires less restrictive assumptions on the structure's topology. In this paper, we address the problem of building a 3-D statistical shape model of the left and right ventricle of the heart from 3-D magnetic resonance images. The average accuracy in landmark propagation is shown to be below 2.2 mm. This application demonstrates the robustness and accuracy of the method in the presence of large shape variability and multiple objects.

An active shape model segmentation scheme is presented that is steered by optimal local features, contrary to normalized first order derivative profiles, as in the original formulation [Cootes and Taylor, 1995, 1999, and 2001]. A nonlinear kNN-classifier is used, instead of the linear Mahalanobis distance, to find optimal displacements for landmarks. For each of the landmarks that describe the shape, at each resolution level taken into account during the segmentation optimization procedure, a distinct set of optimal features is determined. The selection of features is automatic, using the training images and sequential feature forward and backward selection. The new approach is tested on synthetic data and in four medical segmentation tasks: segmenting the right and left lung fields in a database of 230 chest radiographs, and segmenting the cerebellum and corpus callosum in a database of 90 slices from MRI brain images. In all cases, the new method produces significantly better results in terms of an overlap error measure (p < 0.001 using a paired T-test) than the original active shape model scheme.

Three-dimensional (3-D) imaging of the heart is a rapidly developing area of research in medical imaging. Advances in hardware and methods for fast spatio-temporal cardiac imaging are extending the frontiers of clinical diagnosis and research on cardiovascular diseases. In the last few years, many approaches have been proposed to analyze images and extract parameters of cardiac shape and function from a variety of cardiac imaging modalities. In particular, techniques based on spatio-temporal geometric models have received considerable attention. This paper surveys the literature of two decades of research on cardiac modeling. The contribution of the paper is three-fold: 1) to serve as a tutorial of the field for both clinicians and technologists, 2) to provide an extensive account of modeling techniques in a comprehensive and systematic manner, and 3) to critically review these approaches in terms of their performance and degree of clinical evaluation with respect to the final goal of cardiac functional analysis. From this review it is concluded that whereas 3-D model-based approaches have the capability to improve the diagnostic value of cardiac images, issues as robustness, 3-D interaction, computational complexity and clinical validation still require significant attention.

The decrease in the volume of viable tumor is an indicator for the effect preoperative chemotherapy has on bone tumors. We develop an approach for segmenting dynamic perfusion MR-images into viable tumor, nonviable tumor and healthy tissue. Two cascaded feedforward neural networks are trained to perform the pixel-based segmentation. As features, we use the parameters obtained from a pharmacokinetic model of the tissue perfusion (parametric images). Additional multiscale features that incorporate contextual information are included. Experiments indicate that multiscale blurred versions of the parametric images together with a multiscale formulation of the local image entropy are the most discriminative features.

Quantification of the degree of stenosis or vessel dimensions are important for diagnosis of vascular diseases and planning vascular interventions. Although diagnosis from three-dimensional (3-D) magnetic resonance angiograms (MRA's) is mainly performed on two-dimensional (2-D) maximum intensity projections, automated quantification of vascular segments directly from the 3-D dataset is desirable to provide accurate and objective measurements of the 3-D anatomy. A model-based method for quantitative 3-D MRA is proposed. Linear vessel segments are modeled with a central vessel axis curve coupled to a vessel wall surface. A novel image feature to guide the deformation of the central vessel axis is introduced. Subsequently, concepts of deformable models are combined with knowledge of the physics of the acquisition technique to accurately segment the vessel wall and compute the vessel diameter and other geometrical properties. The method is illustrated and validated on a carotid bifurcation phantom, with ground truth and medical experts as comparisons. Also, results on 3-D time-of-flight (TOF) MRA images of the carotids are shown. The approach is a promising technique to assess several geometrical vascular parameters directly on the source 3-D images, providing an objective mechanism for stenosis grading.