In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovász-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.

Population analysis of brain morphology from magnetic resonance images contributes to the study and understanding of neurological diseases. Such analysis typically involves segmentation of a large set of images and comparisons of these segmentations between relevant subgroups of images (e.g., "normal" versus "diseased"). The images of each subgroup are usually selected in advance in a supervised way based on clinical knowledge. Their segmentations are typically guided by one or more available atlases, assumed to be suitable for the images at hand. We present a data-driven probabilistic framework that simultaneously performs atlas-guided segmentation of a heterogeneous set of brain MR images and clusters the images in homogeneous subgroups, while constructing separate probabilistic atlases for each cluster to guide the segmentation. The main benefits of integrating segmentation, clustering and atlas construction in a single framework are that: 1) our method can handle images of a heterogeneous group of subjects and automatically identifies homogeneous subgroups in an unsupervised way with minimal prior knowledge, 2) the subgroups are formed by automatical detection of the relevant morphological features based on the segmentation, 3) the atlases used by our method are constructed from the images themselves and optimally adapted for guiding the segmentation of each subgroup, and 4) the probabilistic atlases represent the morphological pattern that is specific for each subgroup and expose the groupwise differences between different subgroups. We demonstrate the feasibility of the proposed framework and evaluate its performance with respect to image segmentation, clustering and atlas construction on simulated and real data sets including the publicly available BrainWeb and ADNI data. It is shown that combined segmentation and atlas construction leads to improved segmentation accuracy. Furthermore, it is demonstrated that the clusters generated by our unsupervised framework largely coincide with the clinically determined subgroups in case of disease-specific differences in brain morphology and that the differences between the cluster-specific atlases are in agreement with the expected disease-specific patterns, indicating that our method is capable of detecting the different modes in a population. Our method can thus be seen as a comprehensive image-driven population analysis framework that can contribute to the detection of novel subgroups and distinctive image features, potentially leading to new insights in the brain development and disease.

Research in face recognition has continuously been challenged by extrinsic (head pose, lighting conditions) and intrinsic (facial expression, aging) sources of variability. While many survey papers on face recognition exist, in this paper, we focus on a comparative study of 3-D face recognition under expression variations. As a first contribution, 3-D face databases with expressions are listed, and the most important ones are briefly presented and their complexity is quantified using the iterative closest point (ICP) baseline recognition algorithm. This allows to rank the databases according to their inherent difficulty for face-recognition tasks. This analysis reveals that the FRGC v2 database can be considered as the most challenging because of its size, the presence of expressions and outliers, and the time lapse between the recordings. Therefore, we recommend to use this database as a reference database to evaluate (expression-invariant) 3-D face-recognition algorithms. We also determine and quantify the most important factors that influence the performance. It appears that performance decreases 1) with the degree of nonfrontal pose, 2) for certain expression types, 3) with the magnitude of the expressions, 4) with an increasing number of expressions, and 5) for a higher number of gallery subjects. Future 3-D face-recognition algorithms should be evaluated on the basis of all these factors. As the second contribution, a survey of published 3-D face-recognition methods that deal with expression variations is given. These methods are subdivided into three classes depending on the way the expressions are handled. Region-based methods use expression-stable regions only, while other methods model the expressions either using an isometric or a statistical model. Isometric models assume the deformation because of expression variation to be (locally) isometric, meaning that the deformation preserves lengths along the surface. Statistical models learn how the facial soft tissue deforms during expressions based on a training database with expression labels. Algorithmic performances are evaluated by the comparison of recognition rates for identification and verification. No statistical significant differences in class performance are found between any pair of classes.

In this paper the problem of pairwise model-to-scene point set registration is considered. Three contributions are made. Firstly, the relations between correspondence-based and some information-theoretic point cloud registration algorithms are formalized. Starting from the observation that the outlier handling of existing methods relies on heuristically determined models, a second contribution is made exploiting aforementioned relations to derive a new robust point set registration algorithm. Representing model and scene point clouds by mixtures of Gaus-sians, the method minimizes their Kullback-Leibler divergence both w.r.t. the registration transformation parameters and w.r.t. the scene's mixture coefficients. This results in an Expectation-Maximization Iterative Closest Point (EM-ICP) approach with a parameter-free outlier model that is optimal in information-theoretical sense. While the current (CUDA) implementation is limited to the rigid registration case, the underlying theory applies to both rigid and non-rigid point set registration. As a by-product of the registration algorithm's theory, a third contribution is made by suggesting a new point cloud Kernel Density Estimation approach which relies on maximizing the resulting distribution's entropy w.r.t. the kernel weights. The rigid registration algorithm is applied to align different patches of the publicly available Stanford Dragon and Stanford Happy Budha range data. The results show good performance regarding accuracy, robustness and convergence range.

Since most 3D cameras cannot capture the complete 3D face, an important challenge in 3D face recognition is the comparison of two 3D facial surfaces with little or no over lap. In this paper, a local feature method is presented to tackle this challenge exploiting the symmetry of the human face. Features are located and described using an extension of SIFT for meshes (meshSIFT). As such, features are localized as extrema in the curvature scale space of the in put mesh, and are described by concatenating histograms of shape indices and slant angles of the neighborhood. For 3D face scans with sufficient overlap, the number of matching meshSIFT features is a reliable measure for face recognition purposes. However, as the feature descriptor is not symmetrical, features on one face are not matched with their symmetrical counterpart on another face impeding their feasibility for comparison efface scans with limited or no (left-right) overlap. In order to alleviate this problem, facial symmetry could be used to increase the overlap between two face scans by mirroring one of both faces w.r.t. an arbitrary plane. As this would increase the computational demand, this paper proposes an efficient approach to de scribe the features of a mirrored face by mirroring the mesh SIFT descriptors of the input face. The presented method is validated on the data of the "SHREC '11: Face Scans" contest, containing many partial scans. This resulted in a recognition rate of 98.6% and a mean average precision of 93.3%, clearly outperforming all other participants in the challenge.

In this paper a method is presented for the automated identification of cephalometric anatomical landmarks in craniofacial cone-beam CT images. This method makes use of statistical models, incorporating both local appearance and shape knowledge obtained from training data. Firstly, the local appearance model captures the local intensity pattern around each anatomical landmark in the image. Secondly, the shape model contains a local and a global component. The former improves the flexibility, whereas the latter improves the robustness of the algorithm. Using a leave-one-out approach to the training data, we assess the overall accuracy of the method. The mean and median error values for all landmarks are equal to 2.55 mm and 1.72 mm, respectively.

The recognition of faces under varying expressions is one of the current challenges in the face recognition community. In this paper, we propose a method fusing different complementary approaches each dealing with expression variations. The first approach uses an isometric deformation model and is based on the largest singular values of the geodesic distance matrix as an expression-invariant shape descriptor. The second approach performs recognition on the more rigid parts of the face that are less affected by expression variations. Several fusion techniques are examined for combining the approaches. The presented method is validated on a subset of 900 faces of the BU-3DFE face database resulting in an equal error rate of 5.85% for the verification scenario and a rank 1 recognition rate of 94.48% for the identification scenario using the sum rule as fusion technique. This result outperforms other 3D expression-invariant face recognition methods on the same database.

Maximization of mutual information (MMI) is a popular similarity measure for medical image registration. Although its accuracy and robustness has been demonstrated for rigid body image registration, extending MMI to nonrigid image registration is not trivial and an active field of research. We propose conditional mutual information (cMI) as a new similarity measure for nonrigid image registration. cMI starts from a 3-D joint histogram incorporating, besides the intensity dimensions, also a spatial dimension expressing the location of the joint intensity pair. cMI is calculated as the expected value of the cMI between the image intensities given the spatial distribution. The cMI measure was incorporated in a tensor-product B-spline nonrigid registration method, using either a Parzen window or generalized partial volume kernel for histogram construction. cMI was compared to the classical global mutual information (gMI) approach in theoretical, phantom, and clinical settings. We show that cMI significantly outperforms gMI for all applications.

During the registration of 3D CAD models of metallic knee implant components to single-plane sagittal fluoroscopy images, the 3D pose of each implant component is estimated by maximizing the similarity between its 2D image appearance and the observed fluoroscopy image. Because knee implant components are highly symmetrical with respect to the sagittal plane, two significantly different model poses result in 2D image projections very similar to an observed sagittal fluoroscopy image. Traditional 2D/3D registration algorithms tend to converge to one of these symmetrical poses discarding the other one. This paper presents a method which simultaneously estimates both symmetrical solutions. In order not to limit the proposed method to 3D models with an exact plane of symmetry, a completely data-driven symmetry constraint is used which is imposed to the estimated pose parameters. The algorithm is embedded in a statistical framework which is optimized using a deterministic annealing expectation-maximization approach. The validity of the method is demonstrated by registration of the tibial knee implant component to real and simulated fluoroscopy images.

A new generic model-based segmentation algorithm is presented, which can be trained from examples akin to the active shape model (ASM) approach in order to acquire knowledge about the shape to be segmented and about the gray-level appearance of the object in the image. Whereas ASM alternates between shape and intensity information during search, the proposed approach optimizes for shape and intensity characteristics simultaneously. Local gray-level appearance information at the landmark points extracted from feature images is used to automatically detect a number of plausible candidate locations for each landmark. The shape information is described by multiple landmark-specific statistical models that capture local dependencies between adjacent landmarks on the shape. The shape and intensity models are combined in a single cost function that is optimized noniteratively using dynamic programming, without the need for initialization. The algorithm was validated for segmentation of anatomical structures in chest and hand radiographs. In each experiment, the presented method had a significant higher performance when compared to the ASM schemes. As the method is highly effective, optimally suited for pathological cases and easy to implement, it is highly useful for many medical image segmentation tasks.

Analysis of multispectral or multitemporal images requires proper geometric alignment of the images to compare corresponding regions in each image volume. Retrospective three-dimensional alignment or registration of multimodal medical images based on features intrinsic to the image data itself is complicated by their different photometric properties, by the complexity of the anatomical objects in the scene and by the large variety of clinical applications in which registration is involved. While the accuracy of registration approaches based on matching of anatomical landmarks or object surfaces suffers from segmentation errors, voxel-based approaches consider all voxels in the image without the need for segmentation. The recent introduction of the criterion of maximization of mutual information, a basic concept from information theory, has proven to be a breakthrough in the field. While solutions for intrapatient affine registration based on this concept are already commercially available, current research in the field focuses on interpatient nonrigid matching.

Accurate brain tissue segmentation by intensity-based voxel classification of magnetic resonance (MR) images is complicated by partial volume (PV) voxels that contain a mixture of two or more tissue types. In this paper, we present a statistical framework for PV segmentation that encompasses and extends existing techniques. We start from a commonly used parametric statistical image model in which each voxel belongs to one single tissue type, and introduce an additional downsampling step that causes partial voluming along the borders between tissues. An expectation-maximization approach is used to simultaneously estimate the parameters of the resulting model and perform a PV classification. We present results on well-chosen simulated images and on real MR images of the brain, and demonstrate that the use of appropriate spatial prior knowledge not only improves the classifications, but is often indispensable for robust parameter estimation as well. We conclude that general robust PV segmentation of MR brain images requires statistical models that describe the spatial distribution of brain tissues more accurately than currently available models.

We propose a voxel-based nonrigid registration algorithm for temporal subtraction of two-dimensional thorax X-ray computed radiography images of the same subject. The deformation field is represented by a B-spline with a limited number of degrees of freedom, that allows global rib alignment to minimize subtraction artifacts within the lung field without obliterating interval changes of clinically relevant soft-tissue abnormalities. The spline parameters are constrained by a statistical deformation model that is learned from a training set of manually aligned image pairs using principal component analysis. Optimization proceeds along the transformation components rather then along the individual spline coefficients, using pattern intensity of the subtraction image within the automatically segmented lung field region as the criterion to be minimized and applying a simulated annealing strategy for global optimization in the presence of multiple local optima. The impact of different transformation models with varying number of deformation modes is evaluated on a training set of 26 images using a leave-one-out strategy and compared to the manual registration result in terms of criterion value and deformation error. Registration quality is assessed on a second set of validation images by a human expert rating each subtraction image on screen. In 85% of the cases, the registration is subjectively rated to be adequate for clinical use.

This paper presents a fully automated algorithm for segmentation of multiple sclerosis (MS) lesions from multispectral magnetic resonance (MR) images. The method performs intensity-based tissue classification using a stochastic model for normal brain images and simultaneously detects MS lesions as outliers that are not well explained by the model. It corrects for MR field inhomogeneities, estimates tissue-specific intensity models from the data itself, and incorporates contextual information in the classification using a Markov random field. The results of the automated method are compared with lesion delineations by human experts, showing a high total lesion load correlation. When the degree of spatial correspondence between segmentations is taken into account, considerable disagreement is found, both between expert segmentations, and between expert and automatic measurements.

The study presented in this paper tests the hypothesis that the combination of a global similarity transformation and local free-form deformations can be used for the accurate segmentation of internal structures in MR images of the brain. To quantitatively evaluate our approach, the entire brain, the cerebellum, and the head of the caudate have been segmented manually by two raters on one of the volumes (the reference volume) and mapped back onto all the other volumes, using the computed transformations. The contours so obtained have been compared to contours drawn manually around the structures of interest in each individual brain. Manual delineation was performed twice by the same two raters to test inter- and intrarater variability. For the brain and the cerebellum, results indicate that for each rater, contours obtained manually and contours obtained automatically by deforming his own atlas are virtually indistinguishable. Furthermore, contours obtained manually by one rater and contours obtained automatically by deforming this rater's own atlas are more similar than contours obtained manually by two raters. For the caudate, manual intra- and interrater similarity indexes remain slightly better than manual versus automatic indexes, mainly because of the spatial resolution of the images used in this study. Qualitative results also suggest that this method can be used for the segmentation of more complex structures, such as the hippocampus.

We propose a model-based method for fully automated bias field correction of MR brain images. The MR signal is modeled as a realization of a random process with a parametric probability distribution that is corrupted by a smooth polynomial inhomogeneity or bias field. The method we propose applies an iterative expectation-maximization (EM) strategy that interleaves pixel classification with estimation of class distribution and bias field parameters, improving the likelihood of the model parameters at each iteration. The algorithm, which can handle multichannel data and slice-by-slice constant intensity offsets, is initialized with information from a digital brain atlas about the a priori expected location of tissue classes. This allows full automation of the method without need for user interaction, yielding more objective and reproducible results. We have validated the bias correction algorithm on simulated data and we illustrate its performance on various MR images with important field inhomogeneities. We also relate the proposed algorithm to other bias correction algorithms.

We describe a fully automated method for model-based tissue classification of magnetic resonance (MR) images of the brain. The method interleaves classification with estimation of the model parameters, improving the classification at each iteration. The algorithm is able to segment single- and multispectral MR images, corrects for MR signal inhomogeneities, and incorporates contextual information by means of Markov random Fields (MRF's). A digital brain atlas containing prior expectations about the spatial location of tissue classes is used to initialize the algorithm. This makes the method fully automated and therefore it provides objective and reproducible segmentations. We have validated the technique on simulated as well as on real MR images of the brain.

In recent years, advances in computer technology and a significant increase in the accuracy of medical imaging have made it possible to develop systems that can assist the clinician in diagnosis, planning, and treatment. This paper deals with an area that is generally referred to as computer-assisted surgery, image-directed surgery, or image-guided surgery. We report the research, development, and clinical validation performed since January 1996 in the European Applications in Surgical Interventions (EASI) project, which is funded by the European Commission in their "4th Framework Telematics Applications for Health" program. The goal of this project is the improvement of the effectiveness and quality of image-guided neurosurgery of the brain and image-guided vascular surgery of abdominal aortic aneurysms, while at the same time reducing patient risks and overall cost. We have developed advanced prototype systems for preoperative surgical planning and intraoperative surgical navigation, and we have extensively clinically validated these systems. The prototype systems and the clinical validation results are described in this paper.

A new approach to the problem of multimodality medical image registration is proposed, using a basic concept from information theory, mutual information (MI), or relative entropy, as a new matching criterion. The method presented in this paper applies MI to measure the statistical dependence or information redundancy between the image intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. Maximization of MI is a very general and powerful criterion, because no assumptions are made regarding the nature of this dependence and no limiting constraints are imposed on the image content of the modalities involved. The accuracy of the MI criterion is validated for rigid body registration of computed tomography (CT), magnetic resonance (MR), and photon emission tomography (PET) images by comparison with the stereotactic registration solution, while robustness is evaluated with respect to implementation issues, such as interpolation and optimization, and image content, including partial overlap and image degradation. Our results demonstrate that subvoxel accuracy with respect to the stereotactic reference solution can be achieved completely automatically and without any prior segmentation, feature extraction, or other preprocessing steps which makes this method very well suited for clinical applications.

The authors' work contributes to the accurate segmentation of blood vessels from 3D medical images. The blood vessel axis and surface are optimized in an alternating way. Starting from an initial blood vessel axis estimate, slices are resampled in the 3D data volume perpendicular to this axis. In these slices, blood vessel contour candidate points are extracted at maximum gradient positions on a star pattern. The selection of a closed contour among these candidates is optimized with respect to a cost function by dynamic programming. The blood vessel axis is re-estimated at the center of the extracted contours and the process is repeated until convergence. Results are shown on both synthetic and real spiral CT angiographic images.

The objective of this study is to establish a protocol for the technical and clinical evaluation of a workstation for the planning of stereotactic neurosurgical interventions that has been developed in the framework of a joint European research project. Although several such workstations have been proposed before, they lacked the final and most important step, that of clinical validation. They failed to rigorously prove that their product was useful. The authors present a new method that is applicable to the evaluation of a wide range of medical technologies. Their protocol basically assesses the clinical relevance of the user requirements that are at the root of the development of the new technology. The evaluation consists of two stages. During functional specification, iterative prototyping is used to establish the clinical requirements and to assure the quality of the final product. A case study design is used in a second stage that assesses the clinical usability. A before-after study gives a first indication of cost effectiveness and improvement of health care quality.