Segmenting an object in a video presents significant challenges. Each pixel must be accurately labeled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives. The project page is available at https://segment-me-in-time.github.io/

Simplicity bias is a critical challenge in neural net-works since it often leads to favoring simpler solutions and learning unintended decision rules captured by spuri-ous correlations, causing models to be biased and dimin-ishing their generalizability. While existing solutions rely on human supervision, obtaining annotations of the dif-ferent bias attributes is often impractical. To tackle this, we present Debiasify, a novel self-distillation approach that works without any prior information about the nature of biases. Our method leverages a new distillation loss to distill knowledge within a network; from a deep layer where complex, highly-predictive features reside, to a shallow layer where simpler yet attribute-conditioned features are found in an unsupervised manner. In this way, Debiasify learns robust, debiased representations that generalize well across various biases and datasets, enhancing worst-group performance and overall accuracy. Extensive experiments on computer vision and medical imaging benchmarks show the efficacy of our method, significantly outperforming the previous unsupervised debiasing methods (e.g., a 10.13% improvement in worst-group accuracy on Wavy Hair classi-fication in CelebA) while achieving comparable or superior performance to supervised methods. Our code is publicly available at the following link:Debiasify.

Continual Learning (CL) enables neural networks to learn new tasks while retaining previous knowledge. However, most CL methods fail to address bias transfer, where spurious correlations propagate to future tasks or influence past knowledge. This bidirectional bias transfer negatively impacts model performance and fairness, especially in medical imaging, where it can lead to misdiagnoses and unequal treatment. In this work, we show that conventional CL methods amplify these biases, posing risks for diverse patient cohorts. To address this, we propose BiasPruner, a framework that mitigates bias propagation through debiased subnetworks, while preserving sequential learning and avoiding catastrophic forgetting. BiasPruner computes a bias attribution score to identify and prune network units responsible for spurious correlations, creating task-specific subnetworks that learn unbiased representations. As new tasks are learned, the framework integrates non-biased units from previous subnetworks to preserve transferable knowledge and prevent bias transfer. During inference, a task-agnostic gating mechanism selects the optimal subnetwork for robust predictions. We evaluate BiasPruner on medical imaging benchmarks, including skin lesion and chest X-ray classification tasks, where biased data (e.g., spurious skin tone correlations) can exacerbate disparities. Our experiments show that BiasPruner outperforms state-of-the-art CL methods in both accuracy and fairness. Code is available at: BiasPruner.

Significant strides have been made using large vision-language models, like Stable Diffusion (SD), for a variety of downstream tasks, including image generation, image editing, and 3D shape generation. Inspired by these advancements, we explore leveraging these vision-language models for segmenting images at any desired granularity using as few as one annotated sample. We propose SLiMe, which frames this problem as an optimization task. Specifically, given a single image and its segmentation mask, we first extract our novel “weighted accumulated self-attention map” along with cross-attention map from the SD prior. Then, using these extracted maps, the text embeddings of SD are optimized to highlight the segmented region in these attention maps, which in turn can be used to derive new segmentation results. Moreover, leveraging additional training data when available, i.e. few-shot, improves the performance of SLiMe. We performed comprehensive experiments examining various design factors and showed that SLiMe outperforms other existing one-shot and few-shot segmentation methods.

Deep learning models have achieved remarkable success in medical image classification. These models are typically trained once on the available annotated images and thus lack the ability of continually learning new tasks (i.e., new classes or data distributions) due to the problem of catastrophic forgetting. Recently, there has been more interest in designing continual learning methods to learn different tasks presented sequentially over time while preserving previously acquired knowledge. However, these methods focus mainly on preventing catastrophic forgetting and are tested under a closed-world assumption; i.e., assuming the test data is drawn from the same distribution as the training data. In this work, we advance the state-of-the-art in continual learning by proposing GC2 for medical image classification, which learns a sequence of tasks while simultaneously enhancing its out-of-distribution robustness. To alleviate forgetting, GC2 employs a gradual culpability-based network pruning to identify an optimal subnetwork for each task. To improve generalization, GC2 incorporates adversarial image augmentation and knowledge distillation approaches for learning generalized and robust representations for each subnetwork. Our extensive experiments on multiple benchmarks in a task-agnostic inference demonstrate that GC2 significantly outperforms baselines and other continual learning methods in reducing forgetting and enhancing generalization. Our code is publicly available at the following link: https://github.com/nourhanb/TMI2024-GC2.

Being able to explain the prediction to clinical end-users is a necessity to leverage the power of artificial intelligence (AI) models for clinical decision support. For medical images, a feature attribution map, or heatmap, is the most common form of explanation that highlights important features for AI models' prediction. However, it is unknown how well heatmaps perform on explaining decisions on multi-modal medical images, where each image modality or channel visualizes distinct clinical information of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for clinical users' interpretation of AI decisions. To tackle this clinically important but technically ignored problem, we propose the modality-specific feature importance (MSFI) metric. It encodes clinical image and explanation interpretation patterns of modality prioritization and modality-specific feature localization. We conduct a clinical requirement-grounded, systematic evaluation using computational methods and a clinician user study. Results show that the examined 16 heatmap algorithms failed to fulfill clinical requirements to correctly indicate AI model decision process or decision quality. The evaluation and MSFI metric can guide the design and selection of explainable AI algorithms to meet clinical requirements on multi-modal explanation.

Deep learning (DL) models trained to minimize empirical risk on a single domain often fail to generalize when applied to other domains. Model failures due to poor generalizability are quite common in practice and may prove quite perilous in mission-critical applications, e.g., diagnostic imaging where real-world data often exhibits pronounced variability. Such limitations have led to increased interest in domain generalization (DG) approaches that improve the ability of models learned from a single or multiple source domains to generalize to out-of-distribution (OOD) test domains. In this work, we propose BoosterNet, a lean add-on network that can be simply appended to any arbitrary core network to improve its generalization capability without requiring any changes in its architecture or training procedure. Specifically, using a novel measure of feature culpability, BoosterNet is trained episodically on the most and least culpable data features extracted from critical units in the core network based on their contribution towards class-specific prediction errors, which have shown to improve generalization. At inference time, corresponding test image features are extracted from the closest class-specific units, determined by smart gating via a Siamese network, and fed to BoosterNet for improved generalization. We evaluate the performance of BoosterNet within two very different classification problems, digits and skin lesions, and demonstrate a marked improvement in model generalization to OOD test domains compared to SOTA.

A fundamental challenge of over-parameterized deep learning models is learning meaningful data representations that yield good performance on a downstream task without over-fitting spurious input features. This work proposes MaskTune, a masking strategy that prevents over-reliance on spurious (or a limited number of) features. MaskTune forces the trained model to explore new features during a single epoch finetuning by masking previously discovered features. MaskTune, unlike earlier approaches for mitigating shortcut learning, does not require any supervision, such as annotating spurious features or labels for subgroup samples in a dataset. Our empirical results on biased MNIST, CelebA, Waterbirds, and ImagenNet-9L datasets show that MaskTune is effective on tasks that often suffer from the existence of spurious correlations. Finally, we show that \method{} outperforms or achieves similar performance to the competing methods when applied to the selective classification (classification with rejection option) task. Code for MaskTune is available at https://github.com/aliasgharkhani/Masktune.

Drug repurposing can accelerate the identification of effective compounds for clinical use against SARS-CoV-2, with the advantage of pre-existing clinical safety data and an established supply chain. RNA viruses such as SARS-CoV-2 manipulate cellular pathways and induce reorganization of subcellular structures to support their life cycle. These morphological changes can be quantified using bioimaging techniques. In this work, we developed DEEMD: a computational pipeline using deep neural network models within a multiple instance learning framework, to identify putative treatments effective against SARS-CoV-2 based on morphological analysis of the publicly available RxRx19a dataset. This dataset consists of fluorescence microscopy images of SARS-CoV-2 non-infected cells and infected cells, with and without drug treatment. DEEMD first extracts discriminative morphological features to generate cell morphological profiles from the non-infected and infected cells. These morphological profiles are then used in a statistical model to estimate the applied treatment efficacy on infected cells based on similarities to non-infected cells. DEEMD is capable of localizing infected cells via weak supervision without any expensive pixel-level annotations. DEEMD identifies known SARS-CoV-2 inhibitors, such as Remdesivir and Aloxistatin, supporting the validity of our approach. DEEMD can be explored for use on other emerging viruses and datasets to rapidly identify candidate antiviral treatments in the future. Our implementation is available online athttps://www.github.com/Sadegh-Saberian/DEEMD.

Diffuse optical tomography (DOT) leverages near-infrared light propagation through tissue to assess its optical properties and identify abnormalities. DOT image reconstruction is an ill-posed problem due to the highly scattered photons in the medium and the smaller number of measurements compared to the number of unknowns. Limited-angle DOT reduces probe complexity at the cost of increased reconstruction complexity. Reconstructions are thus commonly marred by artifacts and, as a result, it is difficult to obtain an accurate reconstruction of target objects, e.g., malignant lesions. Reconstruction does not always ensure good localization of small lesions. Furthermore, conventional optimization-based reconstruction methods are computationally expensive, rendering them too slow for real-time imaging applications. Our goal is to develop a fast and accurate image reconstruction method using deep learning, where multitask learning ensures accurate lesion localization in addition to improved reconstruction. We apply spatial-wise attention and a distance transform based loss function in a novel multitask learning formulation to improve localization and reconstruction compared to single-task optimized methods. Given the scarcity of real-world sensor-image pairs required for training supervised deep learning models, we leverage physics-based simulation to generate synthetic datasets and use a transfer learning module to align the sensor domain distribution between in silico and real-world data, while taking advantage of cross-domain learning. Applying our method, we find that we can reconstruct and localize lesions faithfully while allowing real-time reconstruction. We also demonstrate that the present algorithm can reconstruct multiple cancer lesions. The results demonstrate that multitask learning provides sharper and more accurate reconstruction.

Kidney volume is an essential biomarker for a number of kidney disease diagnoses, for example, chronic kidney disease. Existing total kidney volume estimation methods often rely on an intermediate kidney segmentation step. On the other hand, automatic kidney localization in volumetric medical images is a critical step that often precedes subsequent data processing and analysis. Most current approaches perform kidney localization via an intermediate classification or regression step. This paper proposes an integrated deep learning approach for (i) kidney localization in computed tomography scans and (ii) segmentation-free renal volume estimation. Our localization method uses a selection-convolutional neural network that approximates the kidney inferior-superior span along the axial direction. Cross-sectional (2D) slices from the estimated span are subsequently used in a combined sagittal-axial Mask-RCNN that detects the organ bounding boxes on the axial and sagittal slices, the combination of which produces a final 3D organ bounding box. Furthermore, we use a fully convolutional network to estimate the kidney volume that skips the segmentation procedure. We also present a mathematical expression to approximate the 'volume error' metric from the 'Sørensen-Dice coefficient.' We accessed 100 patients' CT scans from the Vancouver General Hospital records and obtained 210 patients' CT scans from the 2019 Kidney Tumor Segmentation Challenge database to validate our method. Our method produces a kidney boundary wall localization error of ~2.4mm and a mean volume estimation error of ~5%.

Single molecule localization microscopy (SMLM) allows unprecedented insight into the three-dimensional organization of proteins at the nanometer scale. The combination of minimal invasive cell imaging with high resolution positions SMLM at the forefront of scientific discovery in cancer, infectious, and degenerative diseases. By stochastic temporal and spatial separation of light emissions from fluorescent labelled proteins, SMLM is capable of nanometer scale reconstruction of cellular structures. Precise localization of proteins in 3D astigmatic SMLM is dependent on parameter sensitive preprocessing steps to select regions of interest. With SMLM acquisition highly variable over time, it is non-trivial to find an optimal static parameter configuration. The high emitter density required for reconstruction of complex protein structures can compromise accuracy and introduce artifacts. To address these problems, we introduce two modular auto-tuning pre-processing methods: adaptive signal detection and learned recurrent signal density estimation that can leverage the information stored in the sequence of frames that compose the SMLM acquisition process. We show empirically that our contributions improve accuracy, precision and recall with respect to the state of the art. Both modules auto-tune their hyper-parameters to reduce the parameter space for practitioners, improve robustness and reproducibility, and are validated on a reference in silico dataset. Adaptive signal detection and density prediction can offer a practitioner, in addition to informed localization, a tool to tune acquisition parameters ensuring improved reconstruction of the underlying protein complex. We illustrate the challenges faced by practitioners in applying SMLM algorithms on real world data markedly different from the data used in development and show how ERGO can be run on new datasets without retraining while motivating the need for robust transfer learning in SMLM.

Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However, many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is lost. In this paper, we propose a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan. The proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse sequences and synthesizes the missing ones in a single forward pass. We demonstrate and validate our method on two brain MRI datasets each with four sequences, and show the applicability of the proposed method in simultaneously synthesizing all missing sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach with competing unimodal and multi-modal methods, and show that we outperform both quantitatively and qualitatively.

The linear and non-flexible nature of deep convolutional models makes them vulnerable to carefully crafted adversarial perturbations. To tackle this problem, we propose a non-linear radial basis convolutional feature mapping by learning a Mahalanobis-like distance function. Our method then maps the convolutional features onto a linearly well-separated manifold, which prevents small adversarial perturbations from forcing a sample to cross the decision boundary. We test the proposed method on three publicly available image classification and segmentation datasets namely, MNIST, ISBI ISIC 2017 skin lesion segmentation, and NIH Chest X-Ray-14. We evaluate the robustness of our method to different gradient (targeted and untargeted) and non-gradient based attacks and compare it to several non-gradient masking defense strategies. Our results demonstrate that the proposed method can increase the resilience of deep convolutional neural networks to adversarial perturbations without accuracy drop on clean data.

MOTIVATION: Network analysis and unsupervised machine learning processing of single-molecule localization microscopy of caveolin-1 (Cav1) antibody labeling of prostate cancer cells identified biosignatures and structures for caveolae and three distinct non-caveolar scaffolds (S1A, S1B and S2). To obtain further insight into low-level molecular interactions within these different structural domains, we now introduce graphlet decomposition over a range of proximity thresholds and show that frequency of different subgraph (k = 4 nodes) patterns for machine learning approaches (classification, identification, automatic labeling, etc.) effectively distinguishes caveolae and scaffold blobs. RESULTS: Caveolae formation requires both Cav1 and the adaptor protein CAVIN1 (also called PTRF). As a supervised learning approach, we applied a wide-field CAVIN1/PTRF mask to CAVIN1/PTRF-transfected PC3 prostate cancer cells and used the random forest classifier to classify blobs based on graphlet frequency distribution (GFD). GFD of CAVIN1/PTRF-positive (PTRF+) and -negative Cav1 clusters showed poor classification accuracy that was significantly improved by stratifying the PTRF+ clusters by either number of localizations or volume. Low classification accuracy (<50%) of large PTRF+ clusters and caveolae blobs identified by unsupervised learning suggests that their GFD is specific to caveolae. High classification accuracy for small PTRF+ clusters and caveolae blobs argues that CAVIN1/PTRF associates not only with caveolae but also non-caveolar scaffolds. At low proximity thresholds (50-100 nm), the caveolae groups showed reduced frequency of highly connected graphlets and increased frequency of completely disconnected graphlets. GFD analysis of single-molecule localization microscopy Cav1 clusters defines changes in structural organization in caveolae and scaffolds independent of association with CAVIN1/PTRF. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

We propose a multi-task deep convolutional neural network, trained on multi-modal data (clinical and dermoscopic images, and patient meta-data), to classify the 7-point melanoma checklist criteria and perform skin lesion diagnosis. Our neural network is trained using several multi-task loss functions, where each loss considers different combinations of the input modalities, which allows our model to be robust to missing data at inference time. Our final model classifies the 7-point checklist and skin condition diagnosis, produces multi-modal feature vectors suitable for image retrieval, and localizes clinically discriminant regions. We benchmark our approach using 1011 lesion cases, and report comprehensive results over all 7-point criteria and diagnosis. We also make our dataset (images and metadata) publicly available online at http://derm.cs.sfu.ca.

The presence of certain clinical dermoscopic features within a skin lesion may indicate melanoma, and automatically detecting these features may lead to more quantitative and reproducible diagnoses. We reformulate the task of classifying clinical dermoscopic features within superpixels as a segmentation problem, and propose a fully convolutional neural network to detect clinical dermoscopic features from dermoscopy skin lesion images. Our neural network architecture uses interpolated feature maps from several intermediate network layers, and addresses imbalanced labels by minimizing a negative multilabel Dice-F1score, where the score is computed across the minibatch for each label. Our approach ranked first place in the 2017 ISICISBI Part 2: Dermoscopic Feature Classification Task, challenge over both the provided validation and test datasets, achieving a 0.895% area under the receiver operator characteristic curve score. We show how simple baseline models can outrank state-of-the-art approaches when using the official metrics of the challenge, and propose to use a fuzzy Jaccard Index that ignores the empty set (i.e., masks devoid of positive pixels) when ranking models. Our results suggest that the classification of clinical dermoscopic features can be effectively approached as a segmentation problem, and the current metrics used to rank models may not well capture the efficacy of the model. We plan to make our trained model and code publicly available.

The automatic annotation of Mandarin monosyllabic audio word tokens remains an important yet challenging issue in phonetics research. In this work, we address this annotation task via a novel subcategories-classification framework that not only performs word identification via the joint classifications of vowel and tone subcategories, but also performs gender discrimination of the speaker, which stands in contrast to previously proposed methods for Mandarin speech that focused only on tone-, vowel-, or gender- classification. We also propose a novel hierarchical classification algorithm to boost overall classification performance. Extensive experimental results show that our approach yielded superior performance in both cases of adequate and very limited training data. When trained using data from only one female and one male speaker, our approach also yielded the best classification accuracy in all subcategories of the token annotation problem, achieving an Fl-score of 0.742 as opposed to 0.705 as achieved by the second competing approach.

OBJECTIVE: to provide a proof-of-concept tool for segmenting chronic wounds and transmitting the results as instructions and coordinates to a bioprinter robot and thus facilitate the treatment of chronic wounds. METHODS: several segmentation methods used for measuring wound geometry, including edge-detection and morphological operations, region-growing, Livewire, active contours, and texture segmentation, were compared on 26 images from 15 subjects. Ground-truth wound delineations were generated by a dermatologist. The wound coordinates were converted into G-code understandable by the bioprinting robot. Due to its desirable properties, alginate hydrogel was synthesized by dissolving 16% (w/v) sodium-alginate and 4% (w/v) gelatin in deionized water and used for cell encapsulation. RESULTS: Livewire achieved the best performance, with minimal user interaction: 97.08%, 99.68% 96.67%, 96.22, 98.15, and 32.26, mean values, respectively, for accuracy, sensitivity, specificity, Jaccard index, Dice similarity coefficient, and Hausdorff distance. The bioprinter robot was able to print skin cells on the surface of skin with a 95.56% similarity between the bioprinted patch's dimensions and the desired wound geometry. CONCLUSION: we have designed a novel approach for the healing of chronic wounds, based on semiautomatic segmentation of wound images, improving clinicians' control of the bioprinting process through more accurate coordinates. SIGNIFICANCE: this study is the first to perform wound bioprinting based on image segmentation. It also compares several segmentation methods used for this purpose to determine the best.

It is generally recognized that color information is central to the automatic and visual analysis of histopathology tissue slides. In practice, pathologists rely on color, which reflects the presence of specific tissue components, to establish a diagnosis. Similarly, automatic histopathology image analysis algorithms rely on color or intensity measures to extract tissue features. With the increasing access to digitized histopathology images, color variation and its implications have become a critical issue. These variations are the result of not only a variety of factors involved in the preparation of tissue slides but also in the digitization process itself. Consequently, different strategies have been proposed to alleviate stain-related tissue inconsistencies in automatic image analysis systems. Such techniques generally rely on collecting color statistics to perform color matching across images. In this work, we propose a different approach for stain normalization that we refer to as stain transfer. We design a discriminative image analysis model equipped with a stain normalization component that transfers stains across datasets. Our model comprises a generative network that learns data set-specific staining properties and image-specific color transformations as well as a task-specific network (e.g., classifier or segmentation network). The model is trained end-to-end using a multi-objective cost function. We evaluate the proposed approach in the context of automatic histopathology image analysis on three data sets and two different analysis tasks: tissue segmentation and classification. The proposed method achieves superior results in terms of accuracy and quality of normalized images compared to various baselines.

In this paper, we introduce and evaluate the systems submitted to the first Overlapping Cervical Cytology Image Segmentation Challenge, held in conjunction with the IEEE International Symposium on Biomedical Imaging 2014. This challenge was organized to encourage the development and benchmarking of techniques capable of segmenting individual cells from overlapping cellular clumps in cervical cytology images, which is a prerequisite for the development of the next generation of computer-aided diagnosis systems for cervical cancer. In particular, these automated systems must detect and accurately segment both the nucleus and cytoplasm of each cell, even when they are clumped together and, hence, partially occluded. However, this is an unsolved problem due to the poor contrast of cytoplasm boundaries, the large variation in size and shape of cells, and the presence of debris and the large degree of cellular overlap. The challenge initially utilized a database of 16 high-resolution ( ×40 magnification) images of complex cellular fields of view, in which the isolated real cells were used to construct a database of 945 cervical cytology images synthesized with a varying number of cells and degree of overlap, in order to provide full access of the segmentation ground truth. These synthetic images were used to provide a reliable and comprehensive framework for quantitative evaluation on this segmentation problem. Results from the submitted methods demonstrate that all the methods are effective in the segmentation of clumps containing at most three cells, with overlap coefficients up to 0.3. This highlights the intrinsic difficulty of this challenge and provides motivation for significant future improvement.

In image-guided robotic surgery, segmenting the endoscopic video stream into meaningful parts provides important contextual information that surgeons can exploit to enhance their perception of the surgical scene. This information provides surgeons with real-time decision-making guidance before initiating critical tasks such as tissue cutting. Segmenting endoscopic video is a challenging problem due to a variety of complications including significant noise attributed to bleeding and smoke from cutting, poor appearance contrast between different tissue types, occluding surgical tools, and limited visibility of the objects' geometries on the projected camera views. In this paper, we propose a multi-modal approach to segmentation where preoperative 3D computed tomography scans and intraoperative stereo-endoscopic video data are jointly analyzed. The idea is to segment multiple poorly visible structures in the stereo/multichannel endoscopic videos by fusing reliable prior knowledge captured from the preoperative 3D scans. More specifically, we estimate and track the pose of the preoperative models in 3D and consider the models' non-rigid deformations to match with corresponding visual cues in multi-channel endoscopic video and segment the objects of interest. Further, contrary to most augmented reality frameworks in endoscopic surgery that assume known camera parameters, an assumption that is often violated during surgery due to non-optimal camera calibration and changes in camera focus/zoom, our method embeds these parameters into the optimization hence correcting the calibration parameters within the segmentation process. We evaluate our technique on synthetic data, ex vivo lamb kidney datasets, and in vivo clinical partial nephrectomy surgery with results demonstrating high accuracy and robustness.

We present a novel probabilistic shape representation that implicitly includes prior anatomical volume and adjacency information, termed the generalized log-ratio (GLR) representation. We demonstrate the usefulness of this representation in the task of thigh muscle segmentation. Analysis of the shapes and sizes of thigh muscles can lead to a better understanding of the effects of chronic obstructive pulmonary disease (COPD), which often results in skeletal muscle weakness in lower limbs. However, segmenting these muscles from one another is difficult due to a lack of distinctive features and inter-muscular boundaries that are difficult to detect. We overcome these difficulties by building a shape model in the space of GLR representations. We remove pose variability from the model by employing a presegmentation-based alignment scheme. We also design a rotationally invariant random forest boundary detector that learns common appearances of the interface between muscles from training data. We combine the shape model and the boundary detector into a fully automatic globally optimal segmentation technique. Our segmentation technique produces a probabilistic segmentation that can be used to generate uncertainty information, which can be used to aid subsequent analysis. Our experiments on challenging 3D magnetic resonance imaging data sets show that the use of the GLR representation improves the segmentation accuracy, and yields an average Dice similarity coefficient of 0.808 ±0.074, comparable to other state-of-the-art thigh segmentation techniques.

Cephalometric analysis is an essential clinical and research tool in orthodontics for the orthodontic analysis and treatment planning. This paper presents the evaluation of the methods submitted to the Automatic Cephalometric X-Ray Landmark Detection Challenge, held at the IEEE International Symposium on Biomedical Imaging 2014 with an on-site competition. The challenge was set to explore and compare automatic landmark detection methods in application to cephalometric X-ray images. Methods were evaluated on a common database including cephalograms of 300 patients aged six to 60 years, collected from the Dental Department, Tri-Service General Hospital, Taiwan, and manually marked anatomical landmarks as the ground truth data, generated by two experienced medical doctors. Quantitative evaluation was performed to compare the results of a representative selection of current methods submitted to the challenge. Experimental results show that three methods are able to achieve detection rates greater than 80% using the 4 mm precision range, but only one method achieves a detection rate greater than 70% using the 2 mm precision range, which is the acceptable precision range in clinical practice. The study provides insights into the performance of different landmark detection approaches under real-world conditions and highlights achievements and limitations of current image analysis techniques.

Hair occlusion is one of the main challenges facing automatic lesion segmentation and feature extraction for skin cancer applications. We propose a novel method for simultaneously enhancing both light and dark hairs with variable widths, from dermoscopic images, without the prior knowledge of the hair color. We measure hair tubularness using a quaternion color curvature filter. We extract optimal hair features (tubularness, scale, and orientation) using Markov random field theory and multilabel optimization. We also develop a novel dual-channel matched filter to enhance hair pixels in the dermoscopic images while suppressing irrelevant skin pixels. We evaluate the hair enhancement capabilities of our method on hair-occluded images generated via our new hair simulation algorithm. Since hair enhancement is an intermediate step in a computer-aided diagnosis system for analyzing dermoscopic images, we validate our method and compare it to other methods by studying its effect on: 1) hair segmentation accuracy; 2) image inpainting quality; and 3) image classification accuracy. The validation results on 40 real clinical dermoscopic images and 94 synthetic data demonstrate that our approach outperforms competing hair enhancement methods.

A large number of pigmented skin lesions (PSLs) are a strong predictor of malignant melanoma. Many dermatologists advocate total body photography for high-risk patients because detecting new-appearing, disappearing, and changing PSL is important for early detection of the disease. However, manual inspection and matching of PSL is a subjective, tedious, and error-prone task. A computer program for tracking the corresponding PSL will greatly improve the matching process. In this paper, we describe the construction of the first human back template (atlas), which is used to facilitate spatial normalization of the PSL during the matching process. Four pairs of anatomically meaningful landmarks (neck, shoulder, armpit, and hip points) are used as reference points on the back image. Using the landmarks, a grid with longitudes and latitudes is constructed and overlaid on each subject-specific back image. To perform spatial normalization, the grid is registered into the back template, a unit-square rectilinear grid. To demonstrate the benefits of using the back template, we apply several state-of-the-art point-matching algorithms on 56 pairs of real dermatological images and show that utilizing spatially normalized coordinates improves the PSL matching accuracies.

Sources of uncertainty in the boundaries of structures in medical images have motivated the use of probabilistic labels in segmentation applications. An important component in many medical image segmentation tasks is the use of a shape model, often generated by applying statistical techniques to training data. Standard statistical techniques (e.g., principal component analysis) often assume data lies in an unconstrained vector space, but probabilistic labels are constrained to the unit simplex. If these statistical techniques are used directly on probabilistic labels, relative uncertainty information can be sacrificed. A standard method for facilitating analysis of probabilistic labels is to map them to a vector space using the LogOdds transform. However, the LogOdds transform is asymmetric in one of the labels, which skews results in some applications. The isometric log-ratio (ILR) transform is a symmetrized version of the LogOdds transform, and is so named as it is an isometry between the Aitchison geometry, the inherent geometry of the simplex, and standard Euclidean geometry. We explore how to interpret the Aitchison geometry when applied to probabilistic labels in medical image segmentation applications. We demonstrate the differences when applying the LogOdds transform or the ILR transform to probabilistic labels prior to statistical analysis. Specifically, we show that statistical analysis of ILR transformed data better captures the variability of anatomical shapes in cases where multiple different foreground regions share boundaries (as opposed to foreground-background boundaries).

Incorporating prior knowledge into image segmentation algorithms has proven useful for obtaining more accurate and plausible results. Two important constraints, containment and exclusion of regions, have gained attention in recent years mainly due to their descriptive power. In this paper, we augment the level set framework with the ability to handle these two intuitive geometric relationships, containment and exclusion, along with a distance constraint between boundaries of multi-region objects. Level set's important property of automatically handling topological changes of evolving contours/surfaces enables us to segment spatially-recurring objects (e.g., multiple instances of multi-region cells in a large microscopy image) while satisfying the two aforementioned constraints. In addition, the level set approach gives us a very simple and natural way to compute the distance between contours/surfaces and impose constraints on it. The downside, however, is a local optimization framework in which the final segmentation solution depends on the initialization. In fact, here, we sacrifice the optimizability (local instead of global solution) in exchange for lower space complexity (less memory usage) and faster runtime (especially for large microscopic images) as well as no grid artifacts. Nevertheless, the result from validating our method on several biomedical applications showed the utility and advantages of this augmented level set framework (even with rough initialization that is distant from the desired boundaries). We also compared our framework with its counterpart methods in the discrete domain and reported the pros and cons of each of these methods in terms of metrication error and efficiency in memory usage and runtime.

The inclusion of shape and appearance priors have proven useful for obtaining more accurate and plausible segmentations, especially for complex objects with multiple parts. In this paper, we augment the popular Mum ford-Shah model to incorporate two important geometrical constraints, termed containment and detachment, between different regions with a specified minimum distance between their boundaries. Our method is able to handle multiple instances of multi-part objects defined by these geometrical constraints using a single labeling function while maintaining global optimality. We demonstrate the utility and advantages of these two constraints and show that the proposed convex continuous method is superior to other state-of-the-art methods, including its discrete counterpart, in terms of memory usage, and metrication errors.

Abstract Feature‐driven analysis forms the basis of many shape processing tasks, where detected feature points are characterized by local shape descriptors. Such descriptors have so far been defined to capture regions of interest centred at individual points. Using such regions to compare feature points can be problematic when performing partial shape matching, because the region of interest is typically defined as an isotropic neighbourhood around a point, which does not adapt to the geometry of the shape parts. We introduce the bilateral map, a local shape descriptor whose region of interest is defined by two feature points. Compared to the classical descriptor definition using a single point, the bilateral approach exploits the use of a second point to place more constraints on the selection of the spatial context for feature analysis. This leads to a descriptor where the shape of the region of interest adapts to the context of the two points, making it more refined for shape matching. In particular, we show that our new descriptor is more effective for partial matching, because potentially extraneous regions of the models are selectively ignored owing to the adaptive nature of the bilateral map. This property also renders the bilateral map partially insensitive to topological changes. We demonstrate the effectiveness of the bilateral map for partial matching via several correspondence and retrieval experiments and evaluate the results both qualitatively and quantitatively.

We explore the application of genetic algorithms (GA) to deformable models through the proposition of a novel method for medical image segmentation that combines GA with nonconvex, localized, medial-based shape statistics. We replace the more typical gradient descent optimizer used in deformable models with GA, and the convex, implicit, global shape statistics with nonconvex, explicit, localized ones. Specifically, we propose GA to reduce typical deformable model weaknesses pertaining to model initialization, pose estimation and local minima, through the simultaneous evolution of a large number of models. Furthermore, we constrain the evolution, and thus reduce the size of the search-space, by using statistically-based deformable models whose deformations are intuitive (stretch, bulge, bend) and are driven in terms of localized principal modes of variation, instead of modes of variation across the entire shape that often fail to capture localized shape changes. Although GA are not guaranteed to achieve the global optima, our method compares favorably to the prevalent optimization techniques, convex/nonconvex gradient-based optimizers and to globally optimal graph-theoretic combinatorial optimization techniques, when applied to the task of corpus callosum segmentation in 50 mid-sagittal brain magnetic resonance images.

Noise confounds present serious complications to functional magnetic resonance imaging (fMRI) analysis. The amount of discernible signals within a single dataset of a subject is often inadequate to obtain satisfactory intra-subject activation detection. To remedy this limitation, we propose a novel group Markov random field (GMRF) that extends each subject's neighborhood system to other subjects to enable information coalescing. A distinct advantage of GMRF over standard fMRI group analysis is that no stringent one-to-one voxel correspondence is required. Instead, intra- and inter-subject neighboring voxels are jointly regularized to encourage spatially proximal voxels to be assigned similar labels across subjects. Our proposed group-extended graph structure thus provides an effective means for handling inter-subject variability. Also, adopting a group-wise approach by integrating group information into intra-subject activation, as opposed to estimating a single average group map, permits inter-subject differences to be characterized and studied. GMRF can be elegantly implemented as a single MRF, thus enabling all subjects' activation maps to be simultaneously and collaboratively segmented with global optimality guaranteed in the case of binary labeling. We validate our technique on synthetic and real fMRI data and demonstrate GMRF's superior performance over standard fMRI analysis.

Image segmentation is often performed via the minimization of an energy function over a domain of possible segmentations. The effectiveness and applicability of such methods depends greatly on the properties of the energy function and its domain, and on what information can be encoded by it. Here we propose an energy function that achieves several important goals. Specifically, our energy function is convex and incorporates shape prior information while simultaneously generating a probabilistic segmentation for multiple regions. Our energy function represents multi-region probabilistic segmentations as elements of a vector space using the isometric log-ratio (ILR) transformation. To our knowledge, these four goals (convex, with shape priors, multi-region, and probabilistic) do not exist together in any other method, and this is the first time ILR is used in an image segmentation method. We provide examples demonstrating the usefulness of these features.

Abstract Classical approaches to shape correspondence base their computation purely on the properties, in particular geometric similarity, of the shapes in question. Their performance still falls far short of that of humans in challenging cases where corresponding shape parts may differ significantly in geometry or even topology. We stipulate that in these cases, shape correspondence by humans involves recognition of the shape parts where prior knowledge on the parts would play a more dominant role than geometric similarity. We introduce an approach to part correspondence which incorporates prior knowledge imparted by a training set of pre‐segmented, labeled models and combines the knowledge with content‐driven analysis based on geometric similarity between the matched shapes. First, the prior knowledge is learned from the training set in the form of per‐label classifiers. Next, given two query shapes to be matched, we apply the classifiers to assign a probabilistic label to each shape face. Finally, by means of a joint labeling scheme, the probabilistic labels are used synergistically with pairwise assignments derived from geometric similarity to provide the resulting part correspondence. We show that the incorporation of knowledge is especially effective in dealing with shapes exhibiting large intra‐class variations. We also show that combining knowledge and content analyses outperforms approaches guided by either attribute alone.

Abstract We review methods designed to compute correspondences between geometric shapes represented by triangle meshes, contours or point sets. This survey is motivated in part by recent developments in space–time registration, where one seeks a correspondence between non‐rigid and time‐varying surfaces, and semantic shape analysis, which underlines a recent trend to incorporate shape understanding into the analysis pipeline. Establishing a meaningful correspondence between shapes is often difficult because it generally requires an understanding of the structure of the shapes at both the local and global levels, and sometimes the functionality of the shape parts as well. Despite its inherent complexity, shape correspondence is a recurrent problem and an essential component of numerous geometry processing applications. In this survey, we discuss the different forms of the correspondence problem and review the main solution methods, aided by several classification criteria arising from the problem definition. The main categories of classification are defined in terms of the input and output representation, objective function and solution approach. We conclude the survey by discussing open problems and future perspectives.

Abstract Objects with many concavities are difficult to acquire using laser scanners. The highly concave areas are hard to access by a scanner due to occlusions by other components of the object. The resulting point scan typically suffers from large amounts of missing data. Methods that use surface‐based priors rely on local surface estimates and perform well only when filling small holes. When the holes become large, the reconstruction problem becomes severely under‐constrained, which necessitates the use of additional reconstruction priors. In this paper, we introduce weak volumetric priors which assume that the volume of a shape varies smoothly and that each point cloud sample is visible from outside the shape. Specifically, the union of view‐rays given by the scanner implicitly carves the exterior volume, while volumetric smoothness regularizes the internal volume. We incorporate these priors into a surface evolution framework where a new energy term defined by volumetric smoothness is introduced to handle large amount of missing data. We demonstrate the effectiveness of our method on objects exhibiting deep concavities, and show its general applicability over a broader spectrum of geometric scenario.

A method for visualizing manifold-valued medical image data is proposed. The method operates on images in which each pixel is assumed to be sampled from an underlying manifold. For example, each pixel may contain a high dimensional vector, such as the time activity curve (TAC) in a dynamic positron emission tomography (dPET) or a dynamic single photon emission computed tomography (dSPECT) image, or the positive semi-definite tensor in a diffusion tensor magnetic resonance image (DTMRI). A nonlinear mapping reduces the dimensionality of the pixel data to achieve two goals: distance preservation and embedding into a perceptual color space. We use multidimensional scaling distance-preserving mapping to render similar pixels (e.g., DT or TAC pixels) with perceptually similar colors. The 3D CIELAB perceptual color space is adopted as the range of the distance preserving mapping, with a final similarity transform mapping colors to a maximum gamut size. Similarity between pixels is either determined analytically as geodesics on the manifold of pixels or is approximated using manifold learning techniques. In particular, dissimilarity between DTMRI pixels is evaluated via a Log-Euclidean Riemannian metric respecting the manifold of the rank 3, second-order positive semi-definite DTs, whereas the dissimilarity between TACs is approximated via ISOMAP. We demonstrate our approach via artificial high-dimensional, manifold-valued data, as well as case studies of normal and pathological clinical brain and heart DTMRI, dPET, and dSPECT images. Our results demonstrate the effectiveness of our approach in capturing, in a perceptually meaningful way, important features in the data.

Optical coherence tomography (OCT) is a noninvasive, depth-resolved imaging modality that has become a prominent ophthalmic diagnostic technique. We present a semi-automated segmentation algorithm to detect intra-retinal layers in OCT images acquired from rodent models of retinal degeneration. We adapt Chan-Vese's energy-minimizing active contours without edges for the OCT images, which suffer from low contrast and are highly corrupted by noise. A multiphase framework with a circular shape prior is adopted in order to model the boundaries of retinal layers and estimate the shape parameters using least squares. We use a contextual scheme to balance the weight of different terms in the energy functional. The results from various synthetic experiments and segmentation results on OCT images of rats are presented, demonstrating the strength of our method to detect the desired retinal layers with sufficient accuracy even in the presence of intensity inhomogeneity resulting from blood vessels. Our algorithm achieved an average Dice similarity coefficient of 0.84 over all segmented retinal layers, and of 0.94 for the combined nerve fiber layer, ganglion cell layer, and inner plexiform layer which are the critical layers for glaucomatous degeneration.

Many feature detection algorithms rely on the choice of scale. In this paper, we complement standard scale-selection algorithms with spatial regularization. To this end, we formulate scale-selection as a graph labeling problem and employ Markov random field multi-label optimization. We focus on detecting the scales of vascular structures in medical images. We compare the detected vessel scales using our method to those obtained using the selection approach of the well-known vesselness filter (Frangi et al 1998). We propose and discuss two different approaches for evaluating the goodness of scale-selection. Our results on 40 images from the Digital Retinal Images for Vessel Extraction (DRIVE) database show an average reduction in these error measurements by more than 15%.

Noise confounds present serious complications to accurate data analysis in functional magnetic resonance imaging (fMRI). Simply relying on contextual image information often results in unsatisfactory segmentation of active brain regions. To remedy this, we propose a novel Group Markov Random Field (Group MRF) that extends the neighborhood system to other subjects to incorporate group information in modeling each subject's brain activation. Our approach has the distinct advantage of being able to regularize the states of both intra- and inter-subject neighbors without having to create a stringent one-to-one voxel correspondence as in standard fMRI group analysis. Also, our method can be efficiently implemented as a single MRF, hence enabling activation maps of a group of subjects to be simultaneously and collaboratively segmented. We validate on both synthetic and real fMRI data and demonstrate superior performance over standard analysis techniques.

Abstract In this paper, we develop an interactive analysis and visualization tool for probabilistic segmentation results in medical imaging. We provide a systematic approach to analyze, interact and highlight regions of segmentation uncertainty. We introduce a set of visual analysis widgets integrating different approaches to analyze multivariate probabilistic field data with direct volume rendering. We demonstrate the user's ability to identify suspicious regions (e.g. tumors) and correct the misclassification results using a novel uncertainty‐based segmentation editing technique. We evaluate our system and demonstrate its usefulness in the context of static and time‐varying medical imaging datasets.

Medial representations of shapes are useful due to their use of an object-centered coordinate system that directly captures intuitive notions of shape such as thickness, bending, and elongation. However, it is well known that an object's medial axis transform (MAT) is unstable with respect to small perturbations of its boundary. This instability results in additional, unwanted branches in the skeletons, which must be pruned in order to recover the portions of the skeletons arising purely from the uncorrupted shape information. Almost all approaches to skeleton pruning compute a significance measure for each branch according to some heuristic criteria, and then prune the least significant branches first. Current approaches to branch significance computation can be classified as either local, solely using information from a neighborhood surrounding each branch, or global, using information about the shape as a whole. In this paper, we propose a third, groupwise approach to branch significance computation. We develop a groupwise skeletonization framework that yields a fuzzy significance measure for each branch, derived from information provided by the group of shapes. We call this framework the Groupwise Medial Axis Transform (G-MAT). We propose and evaluate four groupwise methods for computing branch significance and report superior performance compared to a recent, leading method. We measure the performance of each pruning algorithm using denoising, classification, and within-class skeleton similarity measures. This research has several applications, including object retrieval and shape analysis.

We develop an interactive analysis and visualization tool for probabilistic segmentation in medical imaging. The originality of our approach is that the data exploration is guided by shape and appearance knowledge learned from expert-segmented images of a training population. We introduce a set of multidimensional transfer function widgets to analyze the multivariate probabilistic field data. These widgets furnish the user with contextual information about conformance or deviation from the population statistics. We demonstrate the user's ability to identify suspicious regions (e.g. tumors) and to correct the misclassification results. We evaluate our system and demonstrate its usefulness in the context of static anatomical and time-varying functional imaging datasets.

Density of moles is a strong predictor of malignant melanoma. Some dermatologists advocate periodic full-body scan for high-risk patients. In current practice, physicians compare images taken at different time instances to recognize changes. There is an important clinical need to follow changes in the number of moles and their appearance (size, color, texture, shape) in images from two different times. In this paper, we propose a method for finding corresponding moles in patient's skin back images at different scanning times. At first, a template is defined for the human back to calculate the moles' normalized spatial coordinates. Next, matching moles across images is modeled as a graph matching problem and algebraic relations between nodes and edges in the graphs are induced in the matching cost function, which contains terms reflecting proximity regularization, angular agreement between mole pairs, and agreement between the moles' normalized coordinates calculated in the unwarped back template. We propose and discuss alternative approaches for evaluating the goodness of matching. We evaluate our method on a large set of synthetic data (hundreds of pairs) as well as 56 pairs of real dermatological images. Our proposed method compares favorably with the state-of-the-art.

We propose the n-dimensional scale invariant feature transform (n-SIFT) method for extracting and matching salient features from scalar images of arbitrary dimensionality, and compare this method's performance to other related features. The proposed features extend the concepts used for 2-D scalar images in the computer vision SIFT technique for extracting and matching distinctive scale invariant features. We apply the features to images of arbitrary dimensionality through the use of hyperspherical coordinates for gradients and multidimensional histograms to create the feature vectors. We analyze the performance of a fully automated multimodal medical image matching technique based on these features, and successfully apply the technique to determine accurate feature point correspondence between pairs of 3-D MRI images and dynamic 3D + time CT data.

We perform shape matching by transforming the problem of establishing shape correspondences into an image registration problem. At each vertex on the shape, we calculate a shape feature and encode this feature as image intensity at appropriate positions in the image domain. Calculating multiple features at each vertex and encoding them into the image domain results in a vector-valued feature image. Establishing point correspondence between two shapes is thereafter treated as a registration problem of two vector valued feature images. With this shape representation, various existing image registration strategies can now be easily applied. These include the use of a scale-space approach to diffuse the shape features, a coarse-to-fine registration scheme, and various deformable registration algorithms. As our validation shows, by representing shapes as vector valued images, the overall method is robust against noise and occlusions. To this end, we have successfully established 2D point correspondences of shapes of corpora callosa, vertebrae, and brain ventricles.

In this work, we propose and compare several methods for the visualization and exploration of time-varying volumetric medical images based on the temporal characteristics of the data. The principle idea is to consider a time-varying data set as a 3D array where each voxel contains a time-activity curve (TAC). We define and appraise three different TAC similarity measures. Based on these measures we introduce three methods to analyze and visualize time-varying data. The first method relates the whole data set to one template TAC and creates a 1D histogram. The second method extends the 1D histogram into a 2D histogram by taking the Euclidean distance between voxels into account. The third method does not rely on a template TAC but rather creates a 2D scatter-plot of all TAC data points via multi-dimensional scaling. These methods allow the user to specify transfer functions on the 1D and 2D histograms and on the scatter plot, respectively. We validate these methods on synthetic dynamic SPECT and PET data sets and a dynamic planar Gamma camera image of a patient. These techniques are designed to offer researchers and health care professionals a new tool to study the time-varying medical imaging data sets.

We formulate contour correspondence as a Quadratic Assignment Problem (QAP), incorporating proximity information. By maintaining the neighborhood relation between points this way, we show that better matching results are obtained in practice. We propose the first Ant Colony Optimization (ACO) algorithm specifically aimed at solving the QAP-based shape correspondence problem. Our ACO framework is flexible in the sense that it can handle general point correspondence, but also allows extensions, such as order preservation, for the more specialized contour matching problem. Various experiments are presented which demonstrate that this approach yields high-quality correspondence results and is computationally efficient when compared to other methods.

We extend the well-known scalar image bilateral filtering technique to diffusion tensor magnetic resonance images (DTMRI). The scalar version of bilateral image filtering is extended to perform edge-preserving smoothing of DT field data. The bilateral DT filtering is performed in the Log-Euclidean framework which guarantees valid output tensors. Smoothing is achieved by weighted averaging of neighboring tensors. Analogous to bilateral filtering of scalar images, the weights are chosen to be inversely proportional to two distance measures: The geometrical Euclidean distance between the spatial locations of tensors and the dissimilarity of tensors. We describe the noniterative DT smoothing equation in closed form and show how interpolation of DT data is treated as a special case of bilateral filtering where only spatial distance is used. We evaluate different recent DT tensor dissimilarity metrics including the Log-Euclidean, the similarity-invariant Log-Euclidean, the square root of the J-divergence, and the distance scaled mutual diffusion coefficient. We present qualitative and quantitative smoothing and interpolation results and show their effect on segmentation, for both synthetic DT field data, as well as real cardiac and brain DTMRI data.

We present a novel approach to the segmentation and analysis of vasculature from volumetric medical image data. Our method is an adoption and significant extension of deformable organisms, an artificial life framework for medical image analysis that complements classical deformable models with high-level, anatomically-driven control mechanisms. We extend deformable organisms to 3D, model their bodies as tubular spring-mass systems, and equip them with a new repertoire of sensory modules, behavioral routines, and decision making strategies. The result is a new breed of robust deformable organisms, vessel crawlers, that crawl along vasculature in 3D images, accurately segmenting vessel boundaries, detecting and exploring bifurcations, and providing sophisticated, clinically-relevant structural analysis. We validate our method through the segmentation and analysis of vascular structures in both noisy synthetic and real medical image data.

This paper extends 2D Active Shape Models to 2D+time by presenting a method for modelling and segmenting spatio-temporal shapes (ST-shapes). The modelling part consists of constructing a statistical model of ST-shape parameters. The model obtained describes the principal modes of variation of the ST-shape in addition to certain constraints on the allowed variations. An active approach is used in segmentation; an initial ST-shape is deformed to better fit the data and the optimal proposed deformation is calculated using dynamic programming. The results presented show the proposed method detecting ST-shapes in a variety of synthetic noisy data. Preliminary results on real data are also reported. 1

We introduce a new paradigm for automatic medical image analysis that adopts concepts from the field of Artificial Life. Our approach prescribes deformable organisms, autonomous agents whose objective is the segmentation and analysis of anatomical structures in medical images. A deformable organism is structured as a ‘muscle’-actuated ‘body’ whose behavior is controlled by a ‘brain’ that is capable of making both reactive and deliberate decisions. This intelligent deformable model possesses an ‘awareness’ of the segmentation process, which emerges from a conflux of perceived sensory data, an internal mental state, memorized knowledge, and a cognitive plan. We develop a class of deformable organisms using a medial representation of body morphology that facilitates a variety of controlled local deformations at multiple spatial scales. Specifically, we demonstrate a deformable ‘worm‘ organism that can overcome noise, incomplete edges, considerable anatomical variation, and occlusion in order to segment and label the corpus callosum in 2D mid-sagittal MR images of the brain. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper presents the application of active contour models (Snakes) for the segmentation of oral lesions in medical color images acquired from the visual part of the light spectrum. The aim is to assist the clinical expert in locating potentially cancerous cases for further analysis (e.g. classification of cancerous vs. non-cancerous lesions). In order to apply the conventional Snake formulation, color images were converted into single-band images. A number of different single-bands were evaluated including those resulting from the original and normalized RGB, perceptual HSI space, I/sub 1/I/sub 2/I/sub 3/, and the Fisher discriminant function. Examples of Snake segmentation results of oral lesions are presented.

The authors present a method for constraining the deformations of Snakes (active contour models) when segmenting a known class of objects. The method we propose is similar to both active shape models (ASM) but without the landmark identification and correspondence requirement, and to active contour models (ACM), but armed with a priori information about shape variation. Rather than representing the object boundary by spatial landmarks in a point-by-point fashion, we employ a frequency based boundary representation. In this way, the principal component analysis (PCA), which is central to ASM, is applied to a set of frequency-domain shape descriptors, removing the need for the difficult determination of spatial landmarks. Given a training set of representative images of the object of interest, we extract an average object shape along with a set of significant shape variation modes, explaining most of the shape variation in the training set. Armed with this a priori model of shape variation, we find the boundaries in unknown images by placing an initial ACM and allowing it to deform only according to the examined shape variations. The described methodology was applied to a set of 105 echocardiographic images for locating the left ventricular boundary. The results were particularly encouraging in clinically difficult cases where the ventricular boundary was partly occluded by noise.