With an increase in deep learning-based methods, the call for explainability of such methods grows, especially in high-stakes decision making areas such as medical image analysis. This survey presents an overview of explainable artificial intelligence (XAI) used in deep learning-based medical image analysis. A framework of XAI criteria is introduced to classify deep learning-based medical image analysis methods. Papers on XAI techniques in medical image analysis are then surveyed and categorized according to the framework and according to anatomical location. The paper concludes with an outlook of future opportunities for XAI in medical image analysis.

Knowledge of the noise distribution in magnitude diffusion MRI images is the centerpiece to quantify uncertainties arising from the acquisition process. The use of parallel imaging methods, the number of receiver coils and imaging filters applied by the scanner, amongst other factors, dictate the resulting signal distribution. Accurate estimation beyond textbook Rician or noncentral chi distributions often requires information about the acquisition process (e.g., coils sensitivity maps or reconstruction coefficients), which is usually not available. We introduce two new automated methods using the moments and maximum likelihood equations of the Gamma distribution to estimate noise distributions as they explicitly depend on the number of coils, making it possible to estimate all unknown parameters using only the magnitude data. A rejection step is used to make the framework automatic and robust to artifacts. Simulations using stationary and spatially varying noncentral chi noise distributions were created for two diffusion weightings with SENSE or GRAPPA reconstruction and 8, 12 or 32 receiver coils. Furthermore, MRI data of a water phantom with different combinations of parallel imaging were acquired on a 3T Philips scanner along with noise-only measurements. Finally, experiments on freely available datasets from a single subject acquired on a 3T GE scanner are used to assess reproducibility when limited information about the acquisition protocol is available. Additionally, we demonstrated the applicability of the proposed methods for a bias correction and denoising task on an in vivo dataset acquired on a 3T Siemens scanner. A generalized version of the bias correction framework for non integer degrees of freedom is also introduced. The proposed framework is compared with three other algorithms with datasets from three vendors, employing different reconstruction methods. Simulations showed that assuming a Rician distribution can lead to misestimation of the noise distribution in parallel imaging. Results on the acquired datasets showed that signal leakage in multiband can also lead to a misestimation of the noise distribution. Repeated acquisitions of in vivo datasets show that the estimated parameters are stable and have lower variability than compared methods. Results for the bias correction and denoising task show that the proposed methods reduce the appearance of noise at high b-value. The proposed algorithms herein can estimate both parameters of the noise distribution automatically, are robust to signal leakage artifacts and perform best when used on acquired noise maps.

In this study, we propose a fast and accurate method to automatically localize anatomical landmarks in medical images. We employ a global-to-local localization approach using fully convolutional neural networks (FCNNs). First, a global FCNN localizes multiple landmarks through the analysis of image patches, performing regression and classification simultaneously. In regression, displacement vectors pointing from the center of image patches towards landmark locations are determined. In classification, presence of landmarks of interest in the patch is established. Global landmark locations are obtained by averaging the predicted displacement vectors, where the contribution of each displacement vector is weighted by the posterior classification probability of the patch that it is pointing from. Subsequently, for each landmark localized with global localization, local analysis is performed. Specialized FCNNs refine the global landmark locations by analyzing local sub-images in a similar manner, i.e. by performing regression and classification simultaneously and combining the results. Evaluation was performed through localization of 8 anatomical landmarks in CCTA scans, 2 landmarks in olfactory MR scans, and 19 landmarks in cephalometric X-rays. We demonstrate that the method performs similarly to a second observer and is able to localize landmarks in a diverse set of medical images, differing in image modality, image dimensionality, and anatomical coverage.

In patients with obstructive coronary artery disease, the functional significance of a coronary artery stenosis needs to be determined to guide treatment. This is typically established through fractional flow reserve (FFR) measurement, performed during invasive coronary angiography (ICA). We present a method for automatic and non-invasive detection of patients requiring ICA, employing deep unsupervised analysis of complete coronary arteries in cardiac CT angiography (CCTA) images. We retrospectively collected CCTA scans of 187 patients, 137 of them underwent invasive FFR measurement in 192 different coronary arteries. These FFR measurements served as a reference standard for the functional significance of the coronary stenosis. The centerlines of the coronary arteries were extracted and used to reconstruct straightened multi-planar reformatted (MPR) volumes. To automatically identify arteries with functionally significant stenosis that require ICA, each MPR volume was encoded into a fixed number of encodings using two disjoint 3D and 1D convolutional autoencoders performing spatial and sequential encodings, respectively. Thereafter, these encodings were employed to classify arteries using a support vector machine classifier. The detection of coronary arteries requiring invasive evaluation, evaluated using repeated cross-validation experiments, resulted in an area under the receiver operating characteristic curve of 0.81 ± 0.02 on the artery-level, and 0.87 ± 0.02 on the patient-level. The results demonstrate the feasibility of automatic non-invasive detection of patients that require ICA and possibly subsequent coronary artery intervention. This could potentially reduce the number of patients that unnecessarily undergo ICA.

Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. The automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their methods on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge. Sixty T1 + FLAIR images from three MR scanners were released with the manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. The segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: 1) Dice similarity coefficient; 2) modified Hausdorff distance (95th percentile); 3) absolute log-transformed volume difference; 4) sensitivity for detecting individual lesions; and 5) F1-score for individual lesions. In addition, the methods were ranked on their inter-scanner robustness; 20 participants submitted their methods for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all the methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation.

Various types of atherosclerotic plaque and varying grades of stenosis could lead to different management of patients with a coronary artery disease. Therefore, it is crucial to detect and classify the type of coronary artery plaque, as well as to detect and determine the degree of coronary artery stenosis. This paper includes retrospectively collected clinically obtained coronary CT angiography (CCTA) scans of 163 patients. In these, the centerlines of the coronary arteries were extracted and used to reconstruct multi-planar reformatted (MPR) images for the coronary arteries. To define the reference standard, the presence and the type of plaque in the coronary arteries (no plaque, non-calcified, mixed, calcified), as well as the presence and the anatomical significance of coronary stenosis (no stenosis, non-significant, i.e., <50% luminal narrowing, and significant, i.e., ≥50% luminal narrowing) were manually annotated in the MPR images by identifying the start- and end-points of the segment of the artery affected by the plaque. To perform an automatic analysis, a multi-task recurrent convolutional neural network is applied on coronary artery MPR images. First, a 3D convolutional neural network is utilized to extract features along the coronary artery. Subsequently, the extracted features are aggregated by a recurrent neural network that performs two simultaneous multi-class classification tasks. In the first task, the network detects and characterizes the type of the coronary artery plaque. In the second task, the network detects and determines the anatomical significance of the coronary artery stenosis. The network was trained and tested using the CCTA images of 98 and 65 patients, respectively. For detection and characterization of coronary plaque, the method was achieved an accuracy of 0.77. For detection of stenosis and determination of its anatomical significance, the method was achieved an accuracy of 0.80. The results demonstrate that automatic detection and classification of coronary artery plaque and stenosis are feasible. This may enable automated triage of patients to those without coronary plaque and those with coronary plaque and stenosis in need for further cardiovascular workup.

Heavy smokers undergoing screening with low-dose chest CT are affected by cardiovascular disease as much as by lung cancer. Low-dose chest CT scans acquired in screening enable quantification of atherosclerotic calcifications and thus enable identification of subjects at increased cardiovascular risk. This paper presents a method for automatic detection of coronary artery, thoracic aorta, and cardiac valve calcifications in low-dose chest CT using two consecutive convolutional neural networks. The first network identifies and labels potential calcifications according to their anatomical location and the second network identifies true calcifications among the detected candidates. This method was trained and evaluated on a set of 1744 CT scans from the National Lung Screening Trial. To determine whether any reconstruction or only images reconstructed with soft tissue filters can be used for calcification detection, we evaluated the method on soft and medium/sharp filter reconstructions separately. On soft filter reconstructions, the method achieved F1scores of 0.89, 0.89, 0.67, and 0.55 for coronary artery, thoracic aorta, aortic valve, and mitral valve calcifications, respectively. On sharp filter reconstructions, the F1scores were 0.84, 0.81, 0.64, and 0.66, respectively. Linearly weighted kappa coefficients for risk category assignment based on per subject coronary artery calcium were 0.91 and 0.90 for soft and sharp filter reconstructions, respectively. These results demonstrate that the presented method enables reliable automatic cardiovascular risk assessment in all low-dose chest CT scans acquired for lung cancer screening.

Localization of anatomical structures is a prerequisite for many tasks in a medical image analysis. We propose a method for automatic localization of one or more anatomical structures in 3-D medical images through detection of their presence in 2-D image slices using a convolutional neural network (ConvNet). A single ConvNet is trained to detect the presence of the anatomical structure of interest in axial, coronal, and sagittal slices extracted from a 3-D image. To allow the ConvNet to analyze slices of different sizes, spatial pyramid pooling is applied. After detection, 3-D bounding boxes are created by combining the output of the ConvNet in all slices. In the experiments, 200 chest CT, 100 cardiac CT angiography (CTA), and 100 abdomen CT scans were used. The heart, ascending aorta, aortic arch, and descending aorta were localized in chest CT scans, the left cardiac ventricle in cardiac CTA scans, and the liver in abdomen CT scans. Localization was evaluated using the distances between automatically and manually defined reference bounding box centroids and walls. The best results were achieved in the localization of structures with clearly defined boundaries (e.g., aortic arch) and the worst when the structure boundary was not clearly visible (e.g., liver). The method was more robust and accurate in localization multiple structures.

Noise is inherent to low-dose CT acquisition. We propose to train a convolutional neural network (CNN) jointly with an adversarial CNN to estimate routine-dose CT images from low-dose CT images and hence reduce noise. A generator CNN was trained to transform low-dose CT images into routine-dose CT images using voxelwise loss minimization. An adversarial discriminator CNN was simultaneously trained to distinguish the output of the generator from routine-dose CT images. The performance of this discriminator was used as an adversarial loss for the generator. Experiments were performed using CT images of an anthropomorphic phantom containing calcium inserts, as well as patient non-contrast-enhanced cardiac CT images. The phantom and patients were scanned at 20% and 100% routine clinical dose. Three training strategies were compared: the first used only voxelwise loss, the second combined voxelwise loss and adversarial loss, and the third used only adversarial loss. The results showed that training with only voxelwise loss resulted in the highest peak signal-to-noise ratio with respect to reference routine-dose images. However, CNNs trained with adversarial loss captured image statistics of routine-dose images better. Noise reduction improved quantification of low-density calcified inserts in phantom CT images and allowed coronary calcium scoring in low-dose patient CT images with high noise levels. Testing took less than 10 s per CT volume. CNN-based low-dose CT noise reduction in the image domain is feasible. Training with an adversarial network improves the CNNs ability to generate images with an appearance similar to that of reference routine-dose CT images.

Epilepsy is a condition in which periods of ongoing normal EEG activity alternate with periods of oscillatory behavior characteristic of epileptic seizures. The dynamics of the transitions between the two states are still unclear. Computational models provide a powerful tool to explore the underlying mechanisms of such transitions, with the purpose of eventually finding therapeutic interventions for this debilitating condition. In this study, the possibility to postpone seizures elicited by a decrease of inhibition is investigated by using external stimulation in a realistic bistable neuronal model consisting of two interconnected neuronal populations representing pyramidal cells and interneurons. In the simulations, seizures are induced by slowly decreasing the conductivity of GABA[Formula: see text] synaptic channels over time. Since the model is bistable, the system will change state from the initial steady state (SS) to the limit cycle (LS) state because of internal noise, when the inhibition falls below a certain threshold. Several state-independent stimulations paradigms are simulated. Their effectiveness is analyzed for various stimulation frequencies and intensities in combination with periodic and random stimulation sequences. The distributions of the time to first seizure in the presence of stimulation are compared with the situation without stimulation. In addition, stimulation protocols targeted to specific subsystems are applied with the objective of counteracting the baseline shift due to decreased inhibition in the system. Furthermore, an analytical model is used to investigate the effects of random noise. The relation between the strength of random noise stimulation, the control parameter of the system and the transitions between steady state and limit cycle are investigated. The study shows that it is possible to postpone epileptic activity by targeted stimulation in a realistic neuronal model featuring bistability and that it is possible to stop seizures by random noise in an analytical model.

Content-based image retrieval (CBIR) is a search technology that could aid medical diagnosis by retrieving and presenting earlier reported cases that are related to the one being diagnosed. To retrieve relevant cases, CBIR systems depend on supervised learning to map low-level image contents to high-level diagnostic concepts. However, the annotation by medical doctors for training and evaluation purposes is a difficult and time-consuming task, which restricts the supervised learning phase to specific CBIR problems of well-defined clinical applications. This paper proposes a new technique that automatically learns the similarity between the several exams from textual distances extracted from radiology reports, thereby successfully reducing the number of annotations needed. Our method first infers the relation between patients by using information retrieval techniques to determine the textual distances between patient radiology reports. These distances are subsequently used to supervise a metric learning algorithm, that transforms the image space accordingly to textual distances. CBIR systems with different image descriptions and different levels of medical annotations were evaluated, with and without supervision from textual distances, using a database of computer tomography scans of patients with interstitial lung diseases. The proposed method consistently improves CBIR mean average precision, with improvements that can reach 38%, and more marked gains for small annotation sets. Given the overall availability of radiology reports in picture archiving and communication systems, the proposed approach can be broadly applied to CBIR systems in different medical problems, and may facilitate the introduction of CBIR in clinical practice.

Automatic segmentation in MR brain images is important for quantitative analysis in large-scale studies with images acquired at all ages. This paper presents a method for the automatic segmentation of MR brain images into a number of tissue classes using a convolutional neural network. To ensure that the method obtains accurate segmentation details as well as spatial consistency, the network uses multiple patch sizes and multiple convolution kernel sizes to acquire multi-scale information about each voxel. The method is not dependent on explicit features, but learns to recognise the information that is important for the classification based on training data. The method requires a single anatomical MR image only. The segmentation method is applied to five different data sets: coronal T2-weighted images of preterm infants acquired at 30 weeks postmenstrual age (PMA) and 40 weeks PMA, axial T2-weighted images of preterm infants acquired at 40 weeks PMA, axial T1-weighted images of ageing adults acquired at an average age of 70 years, and T1-weighted images of young adults acquired at an average age of 23 years. The method obtained the following average Dice coefficients over all segmented tissue classes for each data set, respectively: 0.87, 0.82, 0.84, 0.86, and 0.91. The results demonstrate that the method obtains accurate segmentations in all five sets, and hence demonstrates its robustness to differences in age and acquisition protocol.

The amount of coronary artery calcification (CAC) is a strong and independent predictor of cardiovascular events. We present a system that automatically quantifies total patient and per coronary artery CAC in non-contrast-enhanced, ECG-triggered cardiac CT. The system identifies candidate calcifications that cannot be automatically labeled with high certainty and optionally presents these to an expert for review. Candidates were extracted by intensity-based thresholding and described by location features derived from estimated coronary artery positions, as well as size, shape and intensity features. Next, a two-class classifier distinguished between coronary calcifications and negatives or a multiclass classifier labeled CAC per coronary artery. Candidates that could not be labeled with high certainty were identified by entropy-based ambiguity detection and presented to an expert for review and possible relabeling. The system was evaluated with 530 test images. Using the two-class classifier, the intra-class correlation coefficient (ICC) between reference and automatically determined total patient CAC volume was 0.95. Using the multiclass classifier, the ICC between reference and automatically determined per artery CAC volume was 0.98 (LAD), 0.69 (LCX), and 0.95 (RCA). In 49% of CTs, no ambiguous candidates were identified, while review of the remaining CTs increased the ICC for total patient CAC volume to 1.00, and per artery CAC volume to 1.00 (LAD), 0.95 (LCX), and 0.99 (RCA). In conclusion, CAC can be automatically identified in non-contrast-enhanced ECG-triggered cardiac CT. Ambiguity detection with expert review may enable the application of automatic CAC scoring in the clinic with a performance comparable to that of a human expert.

In this study, we investigate the correspondence between dynamic patterns of behavior in two types of computational models of neuronal activity. The first model type is the realistic neuronal model; the second model type is the phenomenological or analytical model. In the simplest model set-up of two interconnected units, we define a parameter space for both types of systems where their behavior is similar. Next we expand the analytical model to two sets of 90 fully interconnected units with some overlap, which can display multi-stable behavior. This system can be in three classes of states: (i) a class consisting of a single resting state, where all units of a set are in steady state, (ii) a class consisting of multiple preserving states, where subsets of the units of a set participate in limit cycle, and (iii) a class consisting of a single saturated state, where all units of a set are recruited in a global limit cycle. In the third and final part of the work, we demonstrate that phase synchronization of units can be detected by a single output unit.

We aim to derive fully autonomous seizure suppression paradigms based on reactive control of neuronal dynamics. A previously derived computational model of seizure generation describing collective degrees of freedom and featuring bistable dynamics is used. A novel technique for real-time control of epileptogenicity is introduced. The reactive control reduces practically all seizures in the model. The study indicates which parameters provide the maximal seizure reduction with minimal intervention. An adaptive scheme is proposed that optimizes the stimulation parameters in nonstationary situations.

The calcium burden as estimated from non-ECG-synchronized computed tomography (CT) exams acquired in screening of heavy smokers has been shown to be a strong predictor of cardiovascular events. We present a method for automatic coronary calcium scoring with low-dose, non-contrast-enhanced, non-ECG-synchronized chest CT. First, a probabilistic coronary calcium map was created using multi-atlas segmentation. This map assigned an a priori probability for the presence of coronary calcifications at every location in a scan. Subsequently, a statistical pattern recognition system was designed to identify coronary calcifications by texture, size, and spatial features; the spatial features were computed using the coronary calcium map. The detected calcifications were quantified in terms of volume and Agatston score. The best results were obtained by merging the results of three different supervised classification systems, namely direct classification with a nearest neighbor classifier, and two-stage classification with nearest neighbor and support vector machine classifiers.We used a total of 231 test scans containing 45,674 mm³ of coronary calcifications. The presented method detected on average 157/198 mm³ (sensitivity 79.2%) of coronary calcium volume with on average 4 mm false positive volume. Calcium scoring can be performed automatically in low-dose, non-contrast enhanced, non-ECG-synchronized chest CT in screening of heavy smokers to identify subjects who might benefit from preventive treatment.

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Computer-aided detection (CAD) is increasingly used in clinical practice and for many applications a multitude of CAD systems have been developed. In practice, CAD systems have different strengths and weaknesses and it is therefore interesting to consider their combination. In this paper, we present generic methods to combine multiple CAD systems and investigate what kind of performance increase can be expected. Experimental results are presented using data from the ANODE09 and ROC09 online CAD challenges for the detection of pulmonary nodules in computed tomography scans and red lesions in retinal images, respectively. For both applications, combination results in a large and significant increase in performance when compared to the best individual CAD system.

Medical image registration is an important task in medical image processing. It refers to the process of aligning data sets, possibly from different modalities (e.g., magnetic resonance and computed tomography), different time points (e.g., follow-up scans), and/or different subjects (in case of population studies). A large number of methods for image registration are described in the literature. Unfortunately, there is not one method that works for all applications. We have therefore developed elastix, a publicly available computer program for intensity-based medical image registration. The software consists of a collection of algorithms that are commonly used to solve medical image registration problems. The modular design of elastix allows the user to quickly configure, test, and compare different registration methods for a specific application. The command-line interface enables automated processing of large numbers of data sets, by means of scripting. The usage of elastix for comparing different registration methods is illustrated with three example experiments, in which individual components of the registration method are varied.

In a multi-atlas based segmentation procedure, propagated atlas segmentations must be combined in a label fusion process. Some current methods deal with this problem by using atlas selection to construct an atlas set either prior to or after registration. Other methods estimate the performance of propagated segmentations and use this performance as a weight in the label fusion process. This paper proposes a selective and iterative method for performance level estimation (SIMPLE), which combines both strategies in an iterative procedure. In subsequent iterations the method refines both the estimated performance and the set of selected atlases. For a dataset of 100 MR images of prostate cancer patients, we show that the results of SIMPLE are significantly better than those of several existing methods, including the STAPLE method and variants of weighted majority voting.

A method for automatic segmentation of pulmonary lobes from computed tomography (CT) scans is presented that is robust against incomplete fissures. The method is based on a multiatlas approach in which existing lobar segmentations are deformed to test scans in which the fissures, the lungs, and the bronchial tree have been automatically segmented. The key element of our method is a cost function that exploits information from fissures, lung borders, and bronchial tree in an effective way, such that less reliable information (lungs, airways) is only used when the most reliable information (fissures) is missing. To cope with the anatomical variation in lobe shape, an atlas selection mechanism is introduced. The method is evaluated on two test sets of 120 scans in total. The results show that the lobe segmentation closely follows the fissures when they are present. In a simulated experiment in which parts of complete fissures are removed, the robustness of the method against different levels of incomplete fissures is shown. When the fissures are incomplete, an observer study shows agreement of the automatically determined lobe borders with a radiologist for 81% of the lobe borders on average.

We present a stochastic gradient descent optimisation method for image registration with adaptive step size prediction. The method is based on the theoretical work by Plakhov and Cruz (J. Math. Sci. 120(1):964–973, 2004 ). Our main methodological contribution is the derivation of an image-driven mechanism to select proper values for the most important free parameters of the method. The selection mechanism employs general characteristics of the cost functions that commonly occur in intensity-based image registration. Also, the theoretical convergence conditions of the optimisation method are taken into account. The proposed adaptive stochastic gradient descent (ASGD) method is compared to a standard, non-adaptive Robbins-Monro (RM) algorithm. Both ASGD and RM employ a stochastic subsampling technique to accelerate the optimisation process. Registration experiments were performed on 3D CT and MR data of the head, lungs, and prostate, using various similarity measures and transformation models. The results indicate that ASGD is robust to these variations in the registration framework and is less sensitive to the settings of the user-defined parameters than RM. The main disadvantage of RM is the need for a predetermined step size function. The ASGD method provides a solution for that issue.

A novel atlas-based segmentation approach based on the combination of multiple registrations is presented. Multiple atlases are registered to a target image. To obtain a segmentation of the target, labels of the atlas images are propagated to it. The propagated labels are combined by spatially varying decision fusion weights. These weights are derived from local assessment of the registration success. Furthermore, an atlas selection procedure is proposed that is equivalent to sequential forward selection from statistical pattern recognition theory. The proposed method is compared to three existing atlas-based segmentation approaches, namely 1) single atlas-based segmentation, 2) average-shape atlas-based segmentation, and 3) multi-atlas-based segmentation with averaging as decision fusion. These methods were tested on the segmentation of the heart and the aorta in computed tomography scans of the thorax. The results show that the proposed method outperforms other methods and yields results very close to those of an independent human observer. Moreover, the additional atlas selection step led to a faster segmentation at a comparable performance.

Noise filtering techniques that maintain image contrast while decreasing image noise have the potential to optimize the quality of computed tomography (CT) images acquired at reduced radiation dose. In this paper, a hybrid diffusion filter with continuous switch (HDCS) is introduced, which exploits the benefits of three-dimensional edge-enhancing diffusion (EED) and coherence-enhancing diffusion (CED). Noise is filtered, while edges, tubular structures, and small spherical structures are preserved. From ten high dose thorax CT scans, acquired at clinical doses, ultra low dose ( 15 mAs ) scans were simulated and used to evaluate and compare HDCS to other diffusion filters, such as regularized Perona-Malik diffusion and EED. Quantitative results show that the HDCS filter outperforms the other filters in restoring the high dose CT scan from the corresponding simulated low dose scan. A qualitative evaluation was performed on filtered real low dose CT thorax scans. An expert observer scored artifacts as well as fine structures and was asked to choose one of three scans (two filtered (blinded), one unfiltered) for three different settings (trachea, lung, and mediastinal). Overall, the HDCS filtered scan was chosen most often.

Radiation therapy for cervical cancer can benefit from image registration in several ways, for example by studying the motion of organs, or by (partially) automating the delineation of the target volume and other structures of interest. In this paper, the registration of cervical data is addressed using mutual information (MI) of not only image intensity, but also features that describe local image structure. Three aspects of the registration are addressed to make this approach feasible. First, instead of relying on a histogram-based estimation of mutual information, which poses problems for a larger number of features, a graph-based implementation of alpha-mutual information (alpha-MI) is employed. Second, the analytical derivative of alpha-MI is derived. This makes it possible to use a stochastic gradient descent method to solve the registration problem, which is substantially faster than nonderivative-based methods. Third, the feature space is reduced by means of a principal component analysis, which also decreases the registration time. The proposed technique is compared to a standard approach, based on the mutual information of image intensity only. Experiments are performed on 93 T2-weighted MR clinical data sets acquired from 19 patients with cervical cancer. Several characteristics of the proposed algorithm are studied on a subset of 19 image pairs (one pair per patient). On the remaining data (36 image pairs, one or two pairs per patient) the median overlap is shown to improve significantly compared to standard MI from 0.85 to 0.86 for the clinical target volume (CTV, p = 2 x 10(-2)), from 0.75 to 0.81 for the bladder (p = 8 x 10(-6)), and from 0.76 to 0.77 for the rectum (p = 2 x 10(-4)). The registration error is improved at important tissue interfaces, such as that of the bladder with the CTV, and the interface of the rectum with the uterus and cervix.

This paper presents a new method to enforce inverse consistency in nonrigid image registration and matching. Conventional approaches assume diffeomorphic transformation, implicitly or explicitly. However, the inherent smoothness constraint discourages discontinuity consideration. We propose a post-processing algorithm that integrates the input forward and backward fields, which are output by existing registration/matching algorithms, to produce more robust results. Given such a pair of input fields, our algorithm alternately refines the fields by tensor belief propagation, and enforces inverse consistency in stochastic sense by generalized total least squares fitting. To show the efficacy of our stochastic inverse consistency approach, we first present results on very noisy fields. We then demonstrate improvement on existing stereo matching where occlusion is naturally handled by localizing violations of inverse consistency. Finally, we propose a novel application on image stitching, where stochastic inverse consistency is employed in structure deformation, in order to seamlessly align overlapping images with severe misalignment in structure and intensity.

An approach to 3-D vessel axis tracking based on surface evolution is presented. The main idea is to guide the evolution of the surface by analyzing its skeleton topology during evolution, and imposing shape constraints on the topology. For example, the intermediate topology can be processed such that it represents a single vessel segment, a bifurcation, or a more complex vascular topology. The evolving surface is then reinitialized with the newly found topology. Reinitialization is a crucial step since it creates probing behavior of the evolving front, encourages the segmentation process to extract the vascular structure of interest and reduces the risk on leaking of the curve into the background. The method was evaluated in two computed tomography angiography applications: 1) extracting the internal carotid arteries including the region in which they traverse through the skull base, which is challenging due to the proximity of bone structures and overlap in intensity values; 2) extracting the carotid bifurcations including many cases in which they are severely stenosed and contain calcifications. The vessel axis was found in 90% (18/20 internal carotids in ten patients) and 70% (14/20 carotid bifurcations in a different set of ten patients) of the cases.

A method is presented that uses grouping to improve local classification of image primitives. The grouping process is based upon a spin-glass system, where the image primitives are treated as possessing a spin. The system is subject to an energy functional consisting of a local and a bilocal part, allowing interaction between the image primitives. Instead of defining the state of lowest energy as the grouping result, the mean state of the system is taken. In this way, instabilities caused by multiple minima in the energy are being avoided. The means of the spins are taken as the a posteriori probabilities for the grouping result. In the paper, it is shown how the energy functional can be learned from example data. The energy functional is defined in such a way that, in case of no interactions between the elements, the means of the spins equal the a priori local probabilities. The grouping process enables the fusion of the a priori local and bilocal probabilities into the a posteriori probabilities. The method is illustrated both on grouping of line elements in synthetic images and on vessel detection in retinal fundus images.

A method is introduced to examine the geometrical accuracy of the three-dimensional (3-D) representation of coronary arteries from multiple (two and more) calibrated two-dimensional (2-D) angiographic projections. When involving more then two projections, (multiprojection modeling) a novel procedure is presented that consists of fully automated centerline and width determination in all available projections based on the information provided by the semi-automated centerline detection in two initial calibrated projections. The accuracy of the 3-D coronary modeling approach is determined by a quantitative examination of the 3-D centerline point position and the 3-D cross sectional area of the reconstructed objects. The measurements are based on the analysis of calibrated phantom and calibrated coronary 2-D projection data. From this analysis a confidence region (alpha degrees approximately equal to [35 degrees - 145 degrees]) for the angular distance of two initial projection images is determined for which the modeling procedure is sufficiently accurate for the applied system. Within this angular border range the centerline position error is less then 0.8 mm, in terms of the Euclidean distance to a predefined ground truth. When involving more projections using our new procedure, experiments show that when the initial pair of projection images has an angular distance in the range alpha degrees approximately equal to [35 degrees - 145 degrees], the centerlines in all other projections (gamma = 0 degrees - 180 degrees) were indicated very precisely without any additional centering procedure. When involving additional projection images in the modeling procedure a more realistic shape of the structure can be provided. In case of the concave segment, however, the involvement of multiple projections does not necessarily provide a more realistic shape of the reconstructed structure.

A method is presented to interpolate between neighboring slices in a grey-scale tomographic data set. Spatial correspondence between adjacent slices is established using a nonrigid registration algorithm based on B-splines which optimizes the normalized mutual information similarity measure. Linear interpolation of the image intensities is then carried out along the directions calculated by the registration algorithm. The registration-based method is compared to both standard linear interpolation and shape-based interpolation in 20 tomographic data sets. Results show that the proposed method statistically significantly outperforms both linear and shape-based interpolation.

A measure for registration of medical images that currently draws much attention is mutual information. The measure originates from information theory, but has been shown to be successful for image registration as well. Information theory, however, offers many more measures that may be suitable for image registration. These all measure the divergence of the joint distribution of the images' grey values from the joint distribution that would have been found had the images been completely independent. This paper compares the performance of mutual information as a registration measure with that of other F-information measures. The measures are applied to rigid registration of positron emission tomography (PET)/magnetic resonance (MR) and MR/computed tomography (CT) images, for 35 and 41 image pairs, respectively. An accurate gold standard transformation is available for the images, based on implanted markers. The registration performance, robustness and accuracy of the measures are studied. Some of the measures are shown to perform poorly on all aspects. The majority of measures produces results similar to those of mutual information. An important finding, however, is that several measures, although slightly more difficult to optimize, can potentially yield significantly more accurate results than mutual information.

A method is presented for automated segmentation of vessels in two-dimensional color images of the retina. This method can be used in computer analyses of retinal images, e.g., in automated screening for diabetic retinopathy. The system is based on extraction of image ridges, which coincide approximately with vessel centerlines. The ridges are used to compose primitives in the form of line elements. With the line elements an image is partitioned into patches by assigning each image pixel to the closest line element. Every line element constitutes a local coordinate frame for its corresponding patch. For every pixel, feature vectors are computed that make use of properties of the patches and the line elements. The feature vectors are classified using a kappaNN-classifier and sequential forward feature selection. The algorithm was tested on a database consisting of 40 manually labeled images. The method achieves an area under the receiver operating characteristic curve of 0.952. The method is compared with two recently published rule-based methods of Hoover et al. and Jiang et al. The results show that our method is significantly better than the two rule-based methods (p < 0.01). The accuracy of our method is 0.944 versus 0.947 for a second observer.

A method is presented that uses a vectorial multiscale feature image for wave front propagation between two or more user defined points to retrieve the central axis of tubular objects in digital images. Its implicit scale selection mechanism makes the method more robust to overlap and to the presence of adjacent structures than conventional techniques that propagate a wave front over a scalar image representing the maximum of a range of filters. The method is shown to retain its potential to cope with severe stenoses or imaging artifacts and objects with varying widths in simulated and actual two-dimensional angiographic images.

Using three-dimensional rotational X-ray angiography (3DRA), three-dimensional (3-D) information of the vasculature can be obtained prior to endovascular interventions. However, during interventions, the radiologist has to rely on fluoroscopy images to manipulate the guide wire. In order to take full advantage of the 3-D information from 3DRA data during endovascular interventions, a method is presented that yields an integrated display of the position of the guide wire and vasculature in 3-D. The method relies on an automated method that tracks the guide wire simultaneously in biplane fluoroscopy images. Based on the calibrated geometry of the C-arm, the 3-D guide-wire position is determined and visualized in the 3-D coordinate system of the vasculature. The method is evaluated in an intracranial anthropomorphic vascular phantom. The influence of the angle between projections, distortion correction of the projection images, and accuracy of geometry knowledge on the accuracy of 3-D guide-wire reconstruction from biplane images is determined. If the calibrated geometry information is used and the images are corrected for distortion, a mean distance to the reference standard of 0.42 mm and a tip distance of 0.65 mm is found, which means that accurate guide-wire reconstruction from biplane images can be performed.

A method is presented to extract and track the position of a guide wire during endovascular interventions under X-ray fluoroscopy. The method can be used to improve guide-wire visualization in low-quality fluoroscopic images and to estimate the position of the guide wire in world coordinates. A two-step procedure is utilized to track the guide wire in subsequent frames. First, a rough estimate of the displacement is obtained using a template-matching procedure. Subsequently, the position of the guide wire is determined by fitting a spline to a feature image. The feature images that have been considered enhance line-like structures on: 1) the original images; 2) subtraction images; and 3) preprocessed images in which coherent structures are enhanced. In the optimization step, the influence of the scale at which the feature is calculated and the additional value of using directional information is investigated. The method is evaluated on 267 frames from ten clinical image sequences. Using the automatic method, the guide wire could be tracked in 96% of the frames, with a similar accuracy to three observers, although the position of the tip was estimated less accurately.

Blood pool agents (BPAs) for contrast-enhanced (CE) magnetic-resonance angiography (MRA) allow prolonged imaging times for higher contrast and resolution. Imaging is performed during the steady state when the contrast agent is distributed through the complete vascular system. However, simultaneous venous and arterial enhancement in this steady state hampers interpretation. In order to improve visualization of the arteries and veins from steady-state BPA data, a semiautomated method for artery-vein separation is presented. In this method, the central arterial axis and central venous axis are used as initializations for two surfaces that simultaneously evolve in order to capture the arterial and venous parts of the vasculature using the level-set framework. Since arteries and veins can be in close proximity of each other, leakage from the evolving arterial (venous) surface into the venous (arterial) part of the vasculature is inevitable. In these situations, voxels are labeled arterial or venous based on the arrival time of the respective surface. The evolution is steered by external forces related to feature images derived from the image data and by internal forces related to the geometry of the level sets. In this paper, the robustness and accuracy of three external forces (based on image intensity, image gradient, and vessel-enhancement filtering) and combinations of them are investigated and tested on seven patient datasets. To this end, results with the level-set-based segmentation are compared to the reference-standard manually obtained segmentations. Best results are achieved by applying a combination of intensity- and gradient-based forces and a smoothness constraint based on the curvature of the surface. By applying this combination to the seven datasets, it is shown that, with minimal user interaction, artery-vein separation for improved arterial and venous visualization in BPA CE-MRA can be achieved.

Blood pool agents (BPAs) for contrast-enhanced magnetic resonance angiography (CE-MRA) allow prolonged imaging during the steady state when the agent is distributed through the complete vascular system. This increases both the spatial resolution and the contrast resolution. However, simultaneous venous and arterial enhancement hampers interpretation. For the pelvic region of the vasculature, it is shown that arterial visualization in this equilibrium phase can be improved if the central arterial axis (CAA) is known. However, manually obtaining this axis is not feasible in clinical practice. Therefore, a method is presented that utilizes images acquired during the first pass of the contrast agent to find the CAA in the steady-state data with minimum user initialization. The accuracy of the resulting CAA is compared with tracings of three observers in six patient datasets. It was found that the mean difference between the semiautomatic method and the manual delineation is 1.32 mm in the steady-state data, and that the resulting CAA was always within the arterial lumen, which is an important prerequisite for both improved visualization and segmentation.

A method for localization and segmentation of bifurcated aortic endografts in computed tomographic angiography (CTA) images is presented. The graft position is determined by detecting radiopaque markers sewn on the outside of the graft. The user indicates the first and the last marker, whereupon the remaining markers are automatically detected. This is achieved by first detecting marker-like structures through second-order scaled derivative analysis, which is combined with prior knowledge of graft shape and marker configuration. The identified marker centers approximate the graft sides and, derived from these, the central axis. The graft boundary is determined by maximizing the local gradient in the radial direction along a deformable contour passing through both sides. Three segmentation methods were tested. The first performs graft contour detection in the initial CT-slices, the second in slices that were reformatted to be orthogonal to the approximated graft axis, and the third uses the segmentation from the second method to find a more reliable approximation of the axis and subsequently performs contour detection. The methods have been applied to ten CTA images and the results were compared to manual marker indication by one observer and region growing aided segmentation by three observers. Out of a total of 266 markers, 262 were detected. Adequate approximations of the graft sides were obtained in all cases. The best segmentation results were obtained using a second iteration orthogonal to the axis determined from the first segmentation, yielding an average relative volume of overlap with the expert segmentations of 92%, while the interexpert reproducibility is 95%. The averaged difference in volume measured by the automated method and by the experts equals the difference among the experts: 3.5%.

An overview is presented of the medical image processing literature on mutual-information-based registration. The aim of the survey is threefold: an introduction for those new to the field, an overview for those working in the field, and a reference for those searching for literature on a specific application. Methods are classified according to the different aspects of mutual-information-based registration. The main division is in aspects of the methodology and of the application. The part on methodology describes choices made on facets such as preprocessing of images, gray value interpolation, optimization, adaptations to the mutual information measure, and different types of geometrical transformations. The part on applications is a reference of the literature available on different modalities, on interpatient registration and on different anatomical objects. Comparison studies including mutual information are also considered. The paper starts with a description of entropy and mutual information and it closes with a discussion on past achievements and some future challenges.

This work presents a method to measure the soft tissue motion in three dimensions in the orbit during gaze. It has been shown that two-dimensional (2-D) quantification of soft tissue motion in the orbit is effective in the study of orbital anatomy and motion disorders. However, soft tissue motion is a three-dimensional (3-D) phenomenon and part of the kinematics is lost in any 2-D measurement. Therefore, T1-weighted magnetic resonance (MR) imaging volume sequences are acquired during gaze and soft tissue motion is quantified using a generalization of the Lucas and Kanade optical flow algorithm to three dimensions. New techniques have been developed for visualizing the 3-D flow field as a series of color-texture mapped 2-D slices or as a combination of volume rendering for display of the anatomy and scintillation rendering for the display of the motion field. We have studied the performance of the algorithm on four-dimensional volume sequences of synthetic motion, simulated motion of a static object imaged by MR, an MR-imaged rotating object and MR-imaged motion in the human orbit during gaze. The accuracy of the analysis is sufficient to characterize motion in the orbit and scintillation rendering is an effective visualization technique for 3-D motion in the orbit.

A framework to analyze the propagation of measurement noise through backprojection reconstruction algorithms in electrical impedance tomography (EIT) is presented. Two measurement noise sources were considered: noise in the current drivers and in the voltage detectors. The influence of the acquisition system architecture (serial/semi-parallel) is also discussed. Three variants of backprojection reconstruction are studied: basic (unweighted), weighted and exponential backprojection. The results of error propagation theory have been compared with those obtained from simulated and experimental data. This comparison shows that the approach provides a good estimate of the reconstruction error variance. It is argued that the reconstruction error in EIT images obtained via backprojection can be approximately modeled as a spatially nonstationary Gaussian distribution. This methodology allows us to develop a spatial characterization of the reconstruction error in EIT images.

An active shape model segmentation scheme is presented that is steered by optimal local features, contrary to normalized first order derivative profiles, as in the original formulation [Cootes and Taylor, 1995, 1999, and 2001]. A nonlinear kNN-classifier is used, instead of the linear Mahalanobis distance, to find optimal displacements for landmarks. For each of the landmarks that describe the shape, at each resolution level taken into account during the segmentation optimization procedure, a distinct set of optimal features is determined. The selection of features is automatic, using the training images and sequential feature forward and backward selection. The new approach is tested on synthetic data and in four medical segmentation tasks: segmenting the right and left lung fields in a database of 230 chest radiographs, and segmenting the cerebellum and corpus callosum in a database of 90 slices from MRI brain images. In all cases, the new method produces significantly better results in terms of an overlap error measure (p < 0.001 using a paired T-test) than the original active shape model scheme.

A fully automatic method is presented to detect abnormalities in frontal chest radiographs which are aggregated into an overall abnormality score. The method is aimed at finding abnormal signs of a diffuse textural nature, such as they are encountered in mass chest screening against tuberculosis (TB). The scheme starts with automatic segmentation of the lung fields, using active shape models. The segmentation is used to subdivide the lung fields into overlapping regions of various sizes. Texture features are extracted from each region, using the moments of responses to a multiscale filter bank. Additional "difference features" are obtained by subtracting feature vectors from corresponding regions in the left and right lung fields. A separate training set is constructed for each region. All regions are classified by voting among the k nearest neighbors, with leave-one-out. Next, the classification results of each region are combined, using a weighted multiplier in which regions with higher classification reliability weigh more heavily. This produces an abnormality score for each image. The method is evaluated on two databases. The first database was collected from a TB mass chest screening program, from which 147 images with textural abnormalities and 241 normal images were selected. Although this database contains many subtle abnormalities, the classification has a sensitivity of 0.86 at a specificity of 0.50 and an area under the receiver operating characteristic (ROC) curve of 0.820. The second database consist of 100 normal images and 100 abnormal images with interstitial disease. For this database, the results were a sensitivity of 0.97 at a specificity of 0.90 and an area under the ROC curve of 0.986.

We propose a computational method for segmenting topological subdimensional point-sets in scalar images of arbitrary spatial dimensions. The technique is based on calculating the homotopy class defined by the gradient vector in a subdimensional neighborhood around every image point. This neighborhood is defined as the linear envelope spawned over a given subdimensional vector frame. In the simplest case where the rank of this frame is maximal, we obtain a technique for localizing the critical points. We consider, in particular, the important case of frames formed by an arbitrary number of the first largest by absolute value principal directions of the Hessian. The method then segments positive and and negative ridges as well as other types of critical surfaces of different dimensionalities. The signs of the eigenvalues associated to the principal directions provide a natural labeling of the critical subsets. The result, in general, is a constructive definition of a hierarchy of point-sets of different dimensionalities linked by inclusion relations. Because of its explicit computational nature, the method gives a fast way to segment height ridges or edges in different applications. The defined topological point-sets are connected manifolds and, therefore, our method provides a tool for geometrical grouping using only local measurements. We have demonstrated the grouping properties of our construction by presenting two different cases where an extra image coordinate is introduced.

Three-dimensional (3-D) imaging of the heart is a rapidly developing area of research in medical imaging. Advances in hardware and methods for fast spatio-temporal cardiac imaging are extending the frontiers of clinical diagnosis and research on cardiovascular diseases. In the last few years, many approaches have been proposed to analyze images and extract parameters of cardiac shape and function from a variety of cardiac imaging modalities. In particular, techniques based on spatio-temporal geometric models have received considerable attention. This paper surveys the literature of two decades of research on cardiac modeling. The contribution of the paper is three-fold: 1) to serve as a tutorial of the field for both clinicians and technologists, 2) to provide an extensive account of modeling techniques in a comprehensive and systematic manner, and 3) to critically review these approaches in terms of their performance and degree of clinical evaluation with respect to the final goal of cardiac functional analysis. From this review it is concluded that whereas 3-D model-based approaches have the capability to improve the diagnostic value of cardiac images, issues as robustness, 3-D interaction, computational complexity and clinical validation still require significant attention.

The traditional chest radiograph is still ubiquitous in clinical practice, and will likely remain so for quite some time. Yet, its interpretation is notoriously difficult. This explains the continued interest in computer-aided diagnosis for chest radiography. The purpose of this survey is to categorize and briefly review the literature on computer analysis of chest images, which comprises over 150 papers published in the last 30 years. Remaining challenges are indicated and some directions for future research are given.

This paper presents a technique for automating human scoliosis detection by computer based on moiré topographic images of human backs. Scoliosis is a serious disease often suffered by teenagers. For prevention, screening is performed at schools in Japan employing a moiré method in which doctors inspect moiré images of subjects' backs visually. The inspection of a large number of moiré images collected by the school screening causes exhaustion of doctors and leads to misjudgment. Computer-aided diagnosis of scoliosis has, therefore, been requested eagerly by orthopedists. To automate the inspection process, unlike existent three-dimensional techniques, displacement of local centroids is evaluated two-dimensionally between the left-hand side and the right-hand side of the moiré images in the present technique. The technique was applied to real moiré images to draw a distinction between normal and abnormal cases. According to the leave-out method, the entire 120 image data (60 normal and 60 abnormal) were separated into three data sets. The linear discriminant function based on Mahalanobis distance was defined on the two-dimensional feature space employing one of the data sets containing 40 moiré images and classified 80 images in the remaining two sets. The technique finally achieved the average classification rate of 88.3%.

In this paper, the problem of retrospective correction of intensity inhomogeneity in magnetic resonance (MR) images is addressed. A novel model-based correction method is proposed, based on the assumption that an image corrupted by intensity inhomogeneity contains more information than the corresponding uncorrupted image. The image degradation process is described by a linear model, consisting of a multiplicative and an additive component which are modeled by a combination of smoothly varying basis functions. The degraded image is corrected by the inverse of the image degradation model. The parameters of this model are optimized such that the information of the corrected image is minimized while the global intensity statistic is preserved. The method was quantitatively evaluated and compared to other methods on a number of simulated and real MR images and proved to be effective, reliable, and computationally attractive. The method can be widely applied to different types of MR images because it solely uses the information that is naturally present in an image, without making assumptions on its spatial and intensity distribution. Besides, the method requires no preprocessing, parameter setting, nor user interaction. Consequently, the proposed method may be a valuable tool in MR image analysis.

Magnetic resonance imaging (MRI) offers potential advantages over conventional X-ray techniques for guiding and evaluating intravascular interventions. The development of methods to safely and robustly localize and track devices under MRI guidance is mandatory to enable automatic scan plane adaptation so as to exploit the three-dimensional imaging capabilities of the MRI scanner. With regard to the issue of radiofrequency-induced heating, passive approaches to catheter tracking are inherently safe. These techniques visualize intravascular devices by exploiting the susceptibility artifacts associated with the devices. To promote conspicuity, the devices are equipped with paramagnetic markers. This paper introduces a method to enable automatic localization of devices by its ability to recognize markers in two-dimensional MR images. The method requires a coarse segmentation of the vasculature of interest, and consists of two steps. First, it performs a series of postprocessing operations including calculation of the winding number image and of the Laplacian image to detect marker candidates in the image. Second, the device is localized by matching the detected pattern of candidates to the known distance template of the device markers. Results of an animal experiment and of a clinical application are demonstrated. Validation in phantom experiments shows that the method is able to localize the device in 95% of the cases.

We propose a Bayesian grouping approach for recognition and segmentation of large-scale structures representing objects in images. It is based on detection of local image properties, extraction of simple geometrical primitives, and grouping these primitives according to probability rules and prior models. As opposed to the various template matching techniques, our method does not rely on a fixed set of input data to generate the prior with a maximum likelihood. Instead, it selects a list of subsets of the local primitives and finds the optimum set of model priors that maximizes the likelihood of the model samples representing the selected subsets. In contrast with global recognition methods that classify the whole image, our approach aims at solving the recognition task together with the segmentation task. As an illustration we give a medical data example of feature grouping in 2D images involving vessel detection from local ridges.

The decrease in the volume of viable tumor is an indicator for the effect preoperative chemotherapy has on bone tumors. We develop an approach for segmenting dynamic perfusion MR-images into viable tumor, nonviable tumor and healthy tissue. Two cascaded feedforward neural networks are trained to perform the pixel-based segmentation. As features, we use the parameters obtained from a pharmacokinetic model of the tissue perfusion (parametric images). Additional multiscale features that incorporate contextual information are included. Experiments indicate that multiscale blurred versions of the parametric images together with a multiscale formulation of the local image entropy are the most discriminative features.

Describes the use of heuristics in the determination of a minimum cost path between two points in digital images. The application of four different search methods when applied in two and three dimensional digital images is presented and evaluated. Experiments show that the number of nodes that are being addressed in the search process strongly depends on the discriminative power of the feature used. Furthermore it is shown that for a specific application, the use of a simple heuristic function leads to a considerable reduction in the number of evaluated nodes as compared with the traditional unidirectional approach.

Orbital soft-tissue motion analysis aids in the localization and diagnosis of orbital disorders. A technique has been developed to objectively quantify and visualize motion in the orbit during gaze. T1-weighted MR volume sequences are acquired during gaze and soft-tissue motion is quantified using optical flow techniques. The flow field is visualized using color-coding: orientation of the flow vector is coded by hue and magnitude by saturation of the pixel. Current clinical circumstances limit MR image acquisition to short sequences and short acquisition times. The effect of these limitations on the performance of optical flow computation has been studied for four representative optical flow algorithms: on short (nine frames) and long (21 frames) simulated sequences of rotation of a magnetic resonance (MR) imaged object, on short measured MR sequences of controlled rotation of the same object and on short MR sequences of motion in the orbit. On the short simulated and motion-controlled sequences, the Lucas and Kanade algorithm showed the best performance with respect to both accuracy and robustness. These motion estimates were accurate to within 20%. Motion in the orbit ranged between 0.05 and 0.25 mm/degree gaze. Color-coding was found to be attractive as a visualization technique, because it shows both magnitude and orientation of all flow vectors without cluttering.

Mutual information has developed into an accurate measure for rigid and affine monomodality and multimodality image registration. The robustness of the measure is questionable, however. A possible reason for this is the absence of spatial information in the measure. The present paper proposes to include spatial information by combining mutual information with a term based on the image gradient of the images to be registered. The gradient term not only seeks to align locations of high gradient magnitude, but also aims for a similar orientation of the gradients at these locations. Results of combining both standard mutual information as well as a normalized measure are presented for rigid registration of three-dimensional clinical images [magnetic resonance (MR), computed tomography (CT), and positron emission tomography (PET)]. The results indicate that the combined measures yield a better registration function does mutual information or normalized mutual information per se. The registration functions are less sensitive to low sampling resolution, do not contain incorrect global maxima that are sometimes found in the mutual information function, and interpolation-induced local minima can be reduced. These characteristics yield the promise of more robust registration measures. The accuracy of the combined measures is similar to that of mutual information-based methods.

A method is presented which aids the clinician in obtaining quantitative measures and a three-dimensional (3-D) representation of vessels from 3-D angiographic data with a minimum of user interaction. Based on two user defined starting points, an iterative procedure tracks the central vessel axis. During the tracking process, the minimum diameter and a surface rendering of the vessels are computed, allowing for interactive inspection of the vasculature. Applications of the method to CTA, contrast enhanced (CE)-MRA and phase contrast (PC)-MRA images of the abdomen are shown. In all applications, a long stretch of vessels with varying width is tracked, delineated, and visualized, in less than 10 s on a standard clinical workstation.

In direct area-based stereo problems occur if there is a significant local deformation between the two images, and at borders of objects. We propose a correlation method with affine deformable windows in which an optimal window size is selected using model-based reasoning. Two criteria for the window size selection are studied and the resulting algorithms are applied to a random dot pair, an autostereogram and Hoff's fruit pair. The results are found to be both accurate and robust.

In digital subtraction angiography (DSA), patient motion is the primary cause of image quality degradation. The motion correction algorithms developed so far were not sufficiently fast so as to be suitable for integration in a clinical setting. In this paper we describe a new image registration technique for motion artifact reduction in DSA which is fully automatic, effective, and computationally very efficient. Using an image content driven control point selection mechanism and modern graphics hardware for image warping, the algorithm requires less than one second per DSA image (on average). Preliminary experiments on cerebral DSA images illustrate the applicability of the technique.

A classical approach to interpolation of sampled data is polynomial interpolation. However, from the sampling theorem it follows that the ideal approach to interpolation is to convolve the given samples with the sinc function. In this paper we study the properties of the sinc-approximating kernels that can be derived from the Lagrange central interpolation scheme. Both the finite-extent properties and the convergence property are analyzed. The Lagrange central interpolation kernels of up to ninth order are compared to cardinal splines of corresponding orders, both by spectral analysis and by rotation experiments on real-life test-images. It is concluded that cardinal spline interpolation is by far superior.

The reconstruction of images is an important operation in many applications. From sampling theory, it is well known that the sine-function is the ideal interpolation kernel which, however, cannot be used in practice. In order to be able to obtain an acceptable reconstruction, both in terms of computational speed and mathematical precision, it is required to design a kernel that is of finite extent and resembles the sinc-function as much as possible. In this paper, the applicability of the sine-approximating symmetrical piecewise nth-order polynomial kernels is investigated in satisfying these requirements. After the presentation of the general concept, kernels of first, third, fifth and seventh order are derived. An objective, quantitative evaluation of the reconstruction capabilities of these kernels is obtained by analyzing the spatial and spectral behavior using different measures, and by using them to translate, rotate, and magnify a number of real-life test images. From the experiments, it is concluded that while the improvement of cubic convolution over linear interpolation is significant, the use of higher order polynomials only yields marginal improvement.

Quantification of the degree of stenosis or vessel dimensions are important for diagnosis of vascular diseases and planning vascular interventions. Although diagnosis from three-dimensional (3-D) magnetic resonance angiograms (MRA's) is mainly performed on two-dimensional (2-D) maximum intensity projections, automated quantification of vascular segments directly from the 3-D dataset is desirable to provide accurate and objective measurements of the 3-D anatomy. A model-based method for quantitative 3-D MRA is proposed. Linear vessel segments are modeled with a central vessel axis curve coupled to a vessel wall surface. A novel image feature to guide the deformation of the central vessel axis is introduced. Subsequently, concepts of deformable models are combined with knowledge of the physics of the acquisition technique to accurately segment the vessel wall and compute the vessel diameter and other geometrical properties. The method is illustrated and validated on a carotid bifurcation phantom, with ground truth and medical experts as comparisons. Also, results on 3-D time-of-flight (TOF) MRA images of the carotids are shown. The approach is a promising technique to assess several geometrical vascular parameters directly on the source 3-D images, providing an objective mechanism for stenosis grading.

Digital subtraction angiography (DSA) is a well-established modality for the visualization of blood vessels in the human body. A serious disadvantage of this technique, inherent to the subtraction operation, is its sensitivity to patient motion. The resulting artifacts frequently reduce the diagnostic value of the images. Over the past two decades, many solutions to this problem have been put forward. In this paper, we give an overview of the possible types of motion artifacts and the techniques that have been proposed to avoid them. The main purpose of this paper is to provide a detailed review and discussion of retrospective motion correction techniques that have been described in the literature, to summarize the conclusions that can be drawn from these studies, and to provide suggestions for future research.

In MR brain images, segmentation using intensity values is severely limited owing to field inhomogeneities, susceptibility artifacts and partial volume effects. Edge based segmentation methods suffer from spurious edges and gaps in boundaries. A method is presented which combines the advantages of edge based and region based segmentation. First a multiscale image representation, is constructed which favors intratissue diffusion over inter-tissue diffusion by exploiting local contrast. Subsequently a multiscale linking model (the hyperstack) is used to group voxels into a number of segments. This facilitates segmentation of grey matter, white matter and cerebrospinal fluid with minimal user interaction. Using a supervised segmentation, technique and MR simulations of a brain phantom as validation it is shown that the errors are in the order of or smaller than reported in literature.

Nonlinear diffusion filtering in image processing is usually performed with explicit schemes. They are only stable for very small time steps, which leads to poor efficiency and limits their practical use. Based on a discrete nonlinear diffusion scale-space framework we present semi-implicit schemes which are stable for all time steps. These novel schemes use an additive operator splitting (AOS), which guarantees equal treatment of all coordinate axes. They can be implemented easily in arbitrary dimensions, have good rotational invariance and reveal a computational complexity and memory requirement which is linear in the number of pixels. Examples demonstrate that, under typical accuracy requirements, AOS schemes are at least ten times more efficient than the widely used explicit schemes.

In recent years, advances in computer technology and a significant increase in the accuracy of medical imaging have made it possible to develop systems that can assist the clinician in diagnosis, planning, and treatment. This paper deals with an area that is generally referred to as computer-assisted surgery, image-directed surgery, or image-guided surgery. We report the research, development, and clinical validation performed since January 1996 in the European Applications in Surgical Interventions (EASI) project, which is funded by the European Commission in their "4th Framework Telematics Applications for Health" program. The goal of this project is the improvement of the effectiveness and quality of image-guided neurosurgery of the brain and image-guided vascular surgery of abdominal aortic aneurysms, while at the same time reducing patient risks and overall cost. We have developed advanced prototype systems for preoperative surgical planning and intraoperative surgical navigation, and we have extensively clinically validated these systems. The prototype systems and the clinical validation results are described in this paper.

In this paper implicit representations of deformable models for medical image enhancement and segmentation are considered. The advantage of implicit models over classical explicit models is that their topology can be naturally adapted to objects in the scene. A geodesic formulation of implicit deformable models is especially attractive since it has the energy minimizing properties of classical models. The aim of this paper is twofold. First, a modification to the customary geodesic deformable model approach is introduced by considering all the level sets in the image as energy minimizing contours. This approach is used to segment multiple objects simultaneously and for enhancing and segmenting cardiac computed tomography (CT) and magnetic resonance images. Second, the approach is used to effectively compare implicit and explicit models for specific tasks. This shows the complementary character of implicit models since in case of poor contrast boundaries or gaps in boundaries e.g. due to partial volume effects, noise, or motion artifacts, they do not perform well, since the approach is completely data-driven.

Magnetic resonance imaging (MRI) offers potential advantages over conventional X-ray techniques for guiding and evaluating vascular interventions. Image guidance of such interventions via passive catheter tracking requires real-time image processing. Commercially available MR scanners currently do not provide this functionality. This paper describes an image processing environment that allows near-real-time MR-guided vascular interventions. It demonstrates 1) that flexibility can be achieved by separating the scanner and the image processing/display system, thereby preserving the stability of the scanner and 2) that sufficiently rapid visualization can be achieved by low-cost workstations equipped with graphics hardware. The setup of the hardware and the software is described in detail. Furthermore, image processing techniques are presented for guiding the interventionalist through simple vascular anatomy. Finally, results of a phantom balloon angioplasty experiment are presented.

Coil embolization is a safe treatment for cerebral aneurysms only if the width of the neck in relation to the fundus of the aneurysm is small. Therefore, accurate visualization of the aneurysmal neck is required both in the diagnostic process and during the intervention. Conventional digital subtraction angiography (DSA) is still the preferred modality for the examination of cerebrovascular abnormalities like aneurysms, but it often does not provide the required morphological characteristics due to the suboptimal selection of projection angles and resulting overprojections of surrounding vasculature. This paper presents a method for performing a computer-assisted calculation of the optimal projection angles for DSA by post-processing computed tomographic angiography (CTA) volume data using ray-casting techniques and a combination of image processing algorithms. By means of phantom studies, retrospective simulations of angiograms, and in vivo applications of calculated optimal viewing angles, it is demonstrated that the proposed method results in better angiographic projections of the neck of saccular aneurysms with small neck-fundus ratio than those acquired at standard angles prescribed by clinical protocols.

A hierarchical segmentation model is obtained by using linear scale evolution of gray-scale images. At each scale segments are generated as Voronoi diagrams with a distance measure defined on the image landscape. The set of centers of the Voronoi cells is the set of local extrema of the gray-scale image. This set is localized by using the winding number distribution of the gradient vector field. Scale evolution induces hierarchical structure of embedded segments. Objects defined at coarser scales "decompose" into sub-objects at finer scales. The process is naturally described in terms of singularity catastrophes within the smooth scale evolution. Alternatively, we present a purely topological segmentation procedure, based on singular isophotes. The last are generated by the set of saddle points in the image which are detected also with the topological winding-number method.

In optic flow based velocity estimation the image brightness constraint equation is used. However, for measurements performed at a certain scale, the brightness constraint equation does not apply. We therefore use a recently developed approach which reconciles optic flow and scale space theory. It specifically incorporates the scale (aperture) of image measurements, leading to a scheme which is essentially different from existing approaches. To obtain a unique velocity field, the data-derived information has to be augmented with physical knowledge. By keeping a strict separation between data-derived and external information, we can locally adapt or modify the user-supplied information without affecting the image-derived information. The two free scale parameters in time and space can be used for attentive vision (selecting particular velocities or objects) and to improve the reliability of velocity estimates.

A method is presented to segment multidimensional images using a multiscale (hyperstack) approach with probabilistic linking. A hyperstack is a voxel-based multiscale data structure whose levels are constructed by convolving the original image with a Gaussian kernel of increasing width. Between voxels at adjacent scale levels, child-parent linkages are established according to a model-directed linkage scheme. In the resulting tree-like data structure, roots are formed to indicate the most plausible locations in scale space where segments in the original image are represented by a single voxel. The final segmentation is obtained by tracing back the linkages for all roots. The present paper deals with probabilistic (or multiparent) linking. The multiparent linkage structure is translated into a list of probabilities that are indicative of which voxels are partial volume voxels and to which extent. Probability maps are generated to visualize the progress of weak linkages in scale space when going from fine to coarser scale. It is demonstrated that probabilistic linking gives a significantly improved segmentation as compared with conventional (single-parent) linking.

Recent studies show that the presence of fatty lesions in the atherosclerotic vessel wall is a risk factor for acute occlusion of blood vessels. Although fat has a high electrical resistivity, existing impedance catheter systems cannot be used for detection of these lesions because artifacts owing to impedance variations in the extravascular surroundings have a major and irretraceable effect on the measurement. Standard algorithms used in attempt to compensate for these artifacts suffer from severe instability problems. We defined design guidelines to be met by a new impedance catheter system in order to make a robust reconstruction algorithm possible and have built an experimental in travascular impedance catheter (IIC) system according to these guidelines, using a normalized differential measurement procedure. With this IIC, we performed experiments on human iliac arteries from the section ward (fixed specimens), showing that plastic models of arterial fatty lesions (8 mm3) can be detected reliably.

Multiscale approaches are an invaluable tool for image segmentation. A vast amount of research has been devoted to the construction of different multiscale representations of an image. In this paper we use the hyperstack-a multiscale linking model for image segmentation-for an in-depth comparison of four different scale space generators with respect to segmentation results. We consider the linear (Gaussian) scale space both in the spatial and the Fourier domain, the variable conductance diffusion according to the Perona and Malik equation, and the Euclidean shortening flow. We have done experiments on MR images of the brain, for which a gold standard is available. The hyperstack proves to be rather insensitive to the underlying scale space generator.

Many image processing applications require one to solve problems such as denoising with edge enhancement, preprocessing for segmentation, or the completion of interrupted lines. This may be accomplished by applying a suitable nonlinear anisotropic diffusion process to the image. Its diffusion tensor is adapted to the differential structure of the underlying image. Although being image enhancement tools, filters of this class are well-posed. The temporal evolution of the diffusion process creates a scale-space whose causality properties can be understood in a deterministic, stochastic, information theory based and Fourier based way. The well-posedness and scale-space results carry over to the discrete setting, which gives rise to reliable algorithms preserving all properties of the continuous framework.

Ridge-like structures in digital images may be extracted by convolving the images with derivatives of Gaussians. The choice of the convolution operator and of the parameters involved defines a specific ridge image. In this paper, various ridge measures related to isophote curvature are constructed, reviewed, and evaluated with respect to their usability in CT/MRI matching of human brain scans. Construction is initially done using heuristics in two-dimensional images, and then established firmly in a mathematical framework. Attention is paid to the necessity of operator invariance, scale of the operator, extension to three-dimensional images, and relations to isophote and principal curvature. It is shown that one of the ridge measures appears well suited for the purpose of matching, despite the fact that the measure fails to detect ridges in a number of stylized scenes.

The quality and quantitative accuracy of iteratively reconstructed SPECT images improves when better point spread function (PSF) models of the gamma camera are used during reconstruction. Here, inclusion in the PSF model of photon crosstalk between different slices caused by limited gamma camera resolution and scatter is examined. A three-dimensional (3-D) projector back-projector (proback) has been developed which models both the distance dependent detector point spread function and the object shape-dependent scatter point spread function of single photon emission computed tomography (SPECT). A table occupying only a few megabytes of memory is sufficient to represent this scatter model. The contents of this table are obtained by evaluating an analytical expression for object shape-dependent scatter. The proposed approach avoids the huge memory requirements of storing the full transition matrix needed for 3-D reconstruction including object shape-dependent scatter. In addition, the method avoids the need for lengthy Monte Carlo simulations to generate such a matrix. In order to assess the quantitative accuracy of the method, reconstructions of a water filled cylinder containing regions of different activity levels and of simulated 3-D brain projection data have been evaluated for technetium-99m. It is shown that fully 3-D reconstruction including complete detector response and object shape-dependent scatter modeling clearly outperforms simpler methods that lack a complete detector response and/or a complete scatter response model. Fully 3-D scatter correction yields the best quantitation of volumes of interest and the best contrast-to-noise curves.

A fast simulator of SPECT projection data taking into account attenuation, distance dependent detector response, and scatter has been developed, based on an analytical point spread function model. The parameters of the scatter response are obtained from a single line source measurement with a triangular phantom. The simulator is able to include effects of object curvature on the scatter response to a high accuracy. The simulator has been evaluated for homogeneous media by measurements of (99m)Tc point sources placed at different locations in a water-filled cylinder at energy windows of 15% and 20%. The asymmetrical shapes of measured projections of point sources are In excellent agreement with simulations for both energy windows. Scatter-to-primary ratio (SPR) calculations of point sources at different positions in a cylindrical phantom differ not more than a few percent from measurements. The simulator uses just a few megabytes of memory for storing the tables representing the forward model; furthermore, simulation of 60 SPECT projections from a three-dimensional digital brain phantom with 6-mm cubic voxels takes only ten minutes on a standard workstation. Therefore, the simulator could serve as a projector in iterative true 3-D SPECT reconstruction.

Describes an automated approach to register CT and MR brain images. Differential operators in scale space are applied to each type of image data, so as to produce feature images depicting "ridgeness". The resulting CT and MR feature images show similarities which can be used for matching. No segmentation is needed and the method is devoid of human interaction. The matching is accomplished by hierarchical correlation techniques. Results of 2-D and 3-D matching experiments are presented. The correlation function ensures an accurate match even if the scanned volumes to be matched do not completely overlap, or if some of the features in the images are not similar.

A discrete dynamic model for defining contours in 2-D images is developed. The structure of this model is a set of connected vertices. With a minimum of interaction, an initial contour model can be defined, which is then automatically modified by an energy minimizing process. The internal energy of the model depends on local contour curvature, while the external energy is derived from image features. Solutions are presented to avoid undesirable deformation effects, like shrinking and vertex clustering, which are common in existing active contour models. The deformation process stops when a local minimum of the energy function is reached. The final shape of the model is a reproducible approximation of the desired contour. Results of applying the method to computer-generated images, as well as clinical images, are presented.

In recent years several nonlinear diffusion schemes have been introduced. We discuss the numerical implementation of a number of current nonlinear evolution schemes, using the notion of well-posed differentiation by Gaussian kernels. The infinitesimal change of an image when increasing scale depends on the local differential invariants evaluated at the scale of the image considered, i.e. on terms of the local jet (the set of all spatial partial derivatives at that point). All these differential terms can be obtained in a well-posed fashion by a convolution of the original image with the family of the Gaussian and its derivatives. The nonlinear partial differential evaluation can thus be numerically approximated by an iterative calculation of the appropriate terms in the local jet. Examples are given for medical images.>

This paper presents three classes of speed-up techniques for calculating high-quality perspective Maximum Intensity Projection (MIP) images from Magnetic Resonance Angiography (MRA) data. The method of choice is a combination of Ray Acceleration by Distance Coding (RADC) with Closest Vessel Projection (CVP); this technique combines a five- to ten-fold speed improvement with improved visualization of blood vessels.

Two methods, namely, differential pulse code modulation (DPCM) and hierarchical interpolation (HINT), are considered. It is shown that HINT cannot be extended straightforwardly to 3-D images as contrasted with DPCM. A 3-D HINT is therefore proposed which is based on a combination of 2-D and 3-D filters. Both decorrelation methods were applied to three-dimensional computed tomography (CT), magnetic resonance (MR), and single-photon-emission CT (SPECT) images. It was found that a 3-D approach is optimal for some studies, while for other studies 2-D or even 1-D decorrelation performs better. The optimal dimensionality of DPCM is related to the magnitudes of the local correlation coefficients (CCs). However, the nonlocal nature of HINT makes the local correlation coefficients useless as indicators of the dimensionality; a better candidate is the image voxel size. For images with cubic or nearly cubic voxels 3-D HINT is generally optimal. For images in which the slice thickness is large compared to the pixel size a 2-D (intraslice) HINT is best. In general, the increase in efficiency obtained by extending 2-D decorrelation method to 3-D is small.

A validation methodology for image segmentation methods is explored that is based on quality constrained cost analysis. In this methodology a segmentation method is evaluated by the cost reduction it provides relative to the cost of a full-interactive (manual) segmentation. This cost is constrained by a quality threshold, so that less-than-perfect segmentations are allowed. In this way, segmentation methods which are entirely different in nature can be compared objectively. The validation methodology is presented in its most general form, along with an example of its application to the comparison of segmentation methods.>

An algorithm is presented to detect arrow-shaped skin markers that have been designed for matching of CT and MR images. The markers can be located in the images with subslice accuracy, which allows accurate registration even in standard imaging protocols employing thick slices and/or large interslice gaps. In the past, extracting the marker information from the images has been a highly interactive process. The algorithm described only requires a minimal amount of user interaction.>

In the image processing literature many methods to segment 2D and 3D images have been presented. However, relatively little effort has been spent on the validation of the results of these methods. The goal of the paper is to explore a validation methodology that is based on developing a task-directed quality norm that can be used as a constraint in cost analysis. In this methodology a segmentation method is evaluated by the cost reduction it provides relative to the cost of a full-interactive (manual) segmentation. This cost is constrained by a quality threshold, so that less-than-perfect segmentations are allowed. In this way segmentation methods can be compared, which are designed for the same task, but are different of nature.>

Presents a method of multi-resolution image analysis that gives the most concise set of local orthogonally invariant features of 2-dimensional input images. Solving the equivalence problem corresponding to a local jet and the group of orthogonal transformations of the cartesian coordinate frame the authors find a complete and irreducible set of local algebraic invariants that may describe any local orthogonally invariant feature of the jet.>