We propose a novel pairwise distance measure between image keypoint sets, for the purpose of large-scale medical image indexing. Our measure generalizes the Jaccard index to account for soft set equivalence (SSE) between keypoint elements, via an adaptive kernel framework modeling uncertainty in keypoint appearance and geometry. A new kernel is proposed to quantify the variability of keypoint geometry in location and scale. Our distance measure may be estimated between${O}\,{(}{N}^{{\,{2}}}{)}$image pairs in${O}\,{(}{N}\,\text {log}{N}\,{)}$operations via keypoint indexing. Experiments report the first results for the task of predicting family relationships from medical images, using 1010 T1-weighted MRI brain volumes of 434 families including monozygotic and dizygotic twins, siblings and half-siblings sharing 100%-25% of their polymorphic genes. Soft set equivalence and the keypoint geometry kernel improve upon standard hard set equivalence (HSE) and appearance kernels alone in predicting family relationships. Monozygotic twin identification is near 100%, and three subjects with uncertain genotyping are automatically paired with their self-reported families, the first reported practical application of image-based family identification. Our distance measure can also be used to predict group categories, sex is predicted with an AUC = 0.97. Software is provided for efficient fine-grained curation of large, generic image datasets.

We introduce an approach for image segmentation based on sparse correspondences between keypoints in testing and training images. Keypoints represent automatically identified distinctive image locations, where each keypoint correspondence suggests a transformation between images. We use these correspondences to transfer the label maps of entire organs from the training images to the test image. The keypoint transfer algorithm includes three steps: 1) keypoint matching; 2) voting-based keypoint labeling; and 3) keypoint-based probabilistic transfer of organ segmentations. We report segmentation results for abdominal organs in whole-body CT and MRI, as well as in contrast-enhanced CT and MRI. Our method offers a speed-up of about three orders of magnitude in comparison with common multi-atlas segmentation while achieving an accuracy that compares favorably. Moreover, keypoint transfer does not require the registration to an atlas or a training phase. Finally, the method allows for the segmentation of scans with a highly variable field-of-view.

This paper presents a novel set of image texture features generalizing standard grey-level co-occurrence matrices (GLCM) to multimodal image data through joint intensity matrices (JIMs). These are used to predict the survival of glioblastoma multiforme (GBM) patients from multimodal MRI data. The scans of 73 GBM patients from the Cancer Imaging Archive are used in our study. Necrosis, active tumor, and edema/invasion subregions of GBM phenotypes are segmented using the coregistration of contrast-enhanced T1-weighted (CE-T1) images and its corresponding fluid-attenuated inversion recovery (FLAIR) images. Texture features are then computed from the JIM of these GBM subregions and a random forest model is employed to classify patients into short or long survival groups. Our survival analysis identified JIM features in necrotic (e.g., entropy and inverse-variance) and edema (e.g., entropy and contrast) subregions that are moderately correlated with survival time (i.e., Spearman rank correlation of 0.35). Moreover, nine features were found to be associated with GBM survival with a Hazard-ratio range of 0.38-2.1 and a significance level of p < 0.05 following Holm-Bonferroni correction. These features also led to the highest accuracy in a univariate analysis for predicting the survival group of patients, with AUC values in the range of 68-70%. Considering multiple features for this task, JIM features led to significantly higher AUC values than those based on standard GLCMs and gene expression. Furthermore, an AUC of 77.56% with p = 0.003 was achieved when combining JIM, GLCM, and gene expression features into a single radiogenomic signature. In summary, our study demonstrated the usefulness of modeling the joint intensity characteristics of CE-T1 and FLAIR images for predicting the prognosis of patients with GBM.

This paper presents a method for automatically calibrating and assessing the calibration quality of an externally tracked 2-D ultrasound (US) probe by scanning arbitrary, natural tissues, as opposed a specialized calibration phantom as is the typical practice. A generative topic model quantifies the posterior probability of calibration parameters conditioned on local 2-D image features arising from a generic underlying substrate. Auto-calibration is achieved by identifying the maximum a-posteriori image-to-probe transform, and calibration quality is assessed online in terms of the posterior probability of the current image-to-probe transform. Both are closely linked to the 3-D point reconstruction error (PRE) in aligning feature observations arising from the same underlying physical structure in different US images. The method is of practical importance in that it operates simply by scanning arbitrary textured echogenic structures, e.g., in-vivo tissues in the context of the US-guided procedures, without requiring specialized calibration procedures or equipment. Observed data take the form of local scale-invariant features that can be extracted and fit to the model in near real-time. Experiments demonstrate the method on a public data set of in vivo human brain scans of 14 unique subjects acquired in the context of neurosurgery. Online calibration assessment can be performed at approximately 3 Hz for the US images of pixels. Auto-calibration achieves an internal mean PRE of 1.2 mm and a discrepancy of [2 mm, 6 mm] in comparison to the calibration via a standard phantom-based method.

This paper proposes to use texture features extracted from multispectral microscopic images to detect histopathological abnormalities related to colorectal cancer (CRC): stroma (ST), benign hyperplasia (BH), intraepithelial neoplasia (IN) and carcinoma (Ca). Texture features, based on gray-level co-occurrence matrices (GLCM) and discrete wavelets (DW), are obtained from colon biopsy images, captured using 16 different bands of the visible spectrum. A random forest classifier is used to evaluate the usefulness of these texture features, for each spectral band, on the task of discriminating between the four types of abnormal tissue. Preliminary results on the data of 39 CRC patients show that such features, in particular those based on GLCM and Symlet wavelets, can accurately predict the type of CRC tissue (94% accuracy, 88% sensibility and 100% specificity for Symlet features in the 16thspectral band). These results also reveal important differences in the textural information captured in each band, which could be used to develop more efficient procedures for the diagnosis of CRC.

Sparse multivariate regression techniques like Lasso and Elastic Net are among the most popular approaches for the identification of biomarkers related to brain diseases like Alzheimer's. Because they use L1norm to enforce sparsity, these approaches are often sensitive to differences in voxel intensities within the same scan or across subjects. Also, when few samples are available, such approaches can select voxels that are only correlated by chance, leading to disconnected features that do not correspond to any significant brain structure. To address these challenges, we propose a novel sparse regression method that uses the L0norm for sparse regularization, and imposes spatial consistency constraints on the selected features without requiring an atlas of pre-defined regions. This method uses an efficient optimization strategy based on the Alternating Direction Method of Multipliers (ADMM), that can scale to large data matrices. The performance of the proposed method is evaluated using synthetic data and 3429 T1-weighted (MP-RAGE) images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Experimental results show our method to outperform Lasso and Elastic Net regression in the recovery of spatially consistent features corresponding to known neuroimaging biomarkers.

This paper investigates ordinal image description for invariant feature correspondence. Ordinal description is a meta-technique which considers image measurements in terms of their ranks in a sorted array, instead of the measurement values themselves. Rank-ordering normalizes descriptors in a manner invariant under monotonic deformations of the underlying image measurements, and therefore serves as a simple, non-parametric substitute for ad hoc scaling and thresholding techniques currently used. Ordinal description is particularly well-suited for invariant features, as the high dimensionality of state-of-the-art descriptors permits a large number of unique rank-orderings, and the computationally complex step of sorting is only required once after geometrical normalization. Correspondence trials based on a benchmark data set show that in general, rank-ordered SIFT (SIFT-rank) descriptors outperform other state-of-the-art descriptors in terms of precision-recall, including standard SIFT and GLOH.

This paper presents feature-based morphometry (FBM), a new, fully data-driven technique for identifying group-related differences in volumetric imagery. In contrast to most morphometry methods which assume one-to-one correspondence between all subjects, FBM models images as a collage of distinct, localized image features which may not be present in all subjects. FBM thus explicitly accounts for the case where the same anatomical tissue cannot be reliably identified in all subjects due to disease or anatomical variability. A probabilistic model describes features in terms of their appearance, geometry, and relationship to subgroups of a population, and is automatically learned from a set of subject images and group labels. Features identified indicate group-related anatomical structure that can potentially be used as disease biomarkers or as a basis for computer-aided diagnosis. Scale-invariant image features are used, which reflect generic, salient patterns in the image. Experiments validate FBM clinically in the analysis of normal (NC) and Alzheimer's (AD) brain images using the freely available OASIS database. FBM automatically identifies known structural differences between NC and AD subjects in a fully data-driven fashion, and obtains an equal error classification rate of 0.78 on new subjects.

This paper presents a novel framework for detecting, localizing, and classifying faces in terms of visual traits, e.g., sex or age, from arbitrary viewpoints and in the presence of occlusion. All three tasks are embedded in a general viewpoint-invariant model of object class appearance derived from local scale-invariant features, where features are probabilistically quantified in terms of their occurrence, appearance, geometry, and association with visual traits of interest. An appearance model is first learned for the object class, after which a Bayesian classifier is trained to identify the model features indicative of visual traits. The framework can be applied in realistic scenarios in the presence of viewpoint changes and partial occlusion, unlike other techniques assuming data that are single viewpoint, upright, prealigned, and cropped from background distraction. Experimentation establishes the first result for sex classification from arbitrary viewpoints, an equal error rate of 16.3 percent, based on the color FERET database. The method is also shown to work robustly on faces in cluttered imagery from the CMU profile database. A comparison with the geometry-free bag-of-words model shows that geometrical information provided by our framework improves classification. A comparison with support vector machines demonstrates that Bayesian classification results in superior performance.

In this paper, we investigate detection and localization of general 3D object classes by relating local scale-invariant features to a viewpoint invariant reference frame. This can generally be achieved by either a multi-view representation, where features and reference frame are modeled as a collection of distinct views, or by a viewpoint invariant representation, where features and reference frame are modeled independently of viewpoint. We compare multi-view and viewpoint invariant representations trained and tested on the same data, where the viewpoint invariant approach results in fewer false positive detections and higher average precision. We present a new, iterative learning algorithm to determine an optimal viewpoint invariant reference frame from training images in a data-driven manner. The learned optimal reference frame is centrally located with respect to the 3D object class and to image features in a given view, thereby minimizing reference frame localization error as predicted by theory and maintaining a consistent geometrical interpretation with respect to the underlying object class. Modeling and detection based on the optimal reference frame improves detection performance for both multiview and viewpoint invariant representations. Experimentation is performed on the class of 3D faces, using the public color FERET database for training, the CMU profile database for testing and SIFT image features.

In this paper, we present a statistical parts-based model (PBM) of appearance, applied to the problem of modeling intersubject anatomical variability in magnetic resonance (MR) brain images. In contrast to global image models such as the active appearance model (AAM), the PBM consists of a collection of localized image regions, referred to as parts, whose appearance, geometry and occurrence frequency are quantified statistically. The parts-based approach explicitly addresses the case where one-to-one correspondence does not exist between all subjects in a population due to anatomical differences, as model parts are not required to appear in all subjects. The model is constructed through a fully automatic machine learning algorithm, identifying image patterns that appear with statistical regularity in a large collection of subject images. Parts are represented by generic scale-invariant features, and the model can, therefore, be applied to a wide variety of image domains. Experimentation based on 2-D MR slices shows that a PBM learned from a set of 102 subjects can be robustly fit to 50 new subjects with accuracy comparable to 3 human raters. Additionally, it is shown that unlike global models such as the AAM, PBM fitting is stable in the presence of unexpected, local perturbation.

Feature points for image correspondence are often selected according to subjective criteria (e.g. edge density, nostrils). In this paper, we present a general, nonsubjective criterion for selecting informative feature points, based on the correspondence model itself. We describe the approach within the framework of the Bayesian Markov random field (MRF) model, where the degree of feature point information is encoded by the entropy of the likelihood term. We propose that feature selection according to minimum entropy-of-likelihood (EOL) is less likely to lead to correspondence ambiguity, thus improving the optimization process in terms of speed and quality of solution. Experimental results demonstrate the criterion's ability to select optimal features points in a wide variety of image contexts (e.g. objects, faces). Comparison with the automatic Kanade-Lucas-Tomasi feature selection criterion shows correspondence to be significantly faster with feature points selected according to minimum EOL in difficult correspondence problems.