Functional magnetic resonance imaging (fMRI) is a powerful tool for investigating human brain function. However, the high cost of data acquisition and the inherent subjectivity of psychiatric rating scales often lead to datasets with small sample sizes and variable label quality, especially when targeting a specific neurological condition. Combined with the inherently high dimensionality of fMRI data, these limitations substantially increase the risk of model overfitting. Recent years have seen growing interest in developing fMRI foundation models by combining multiple datasets; however, the computational resources needed for pretraining and fine-tuning are often prohibitive. We show that a lightweight self-supervised framework yields representations that generalize across diverse downstream tasks, outperforming fully supervised baselines and approaching the performance of large-scale models. We introduce BrainSimSiam, a data-efficient self-supervised representation learning framework that leverages positive-only data pairs to learn robust and generalizable features. We demonstrate that the learned representations achieve strong performance across multiple downstream classification and regression tasks, highlighting the potential of BrainSimSiam for data-limited neuroimaging applications. Our implementation is available in https://github.com/Jiyao96/BrainSimSiam-fMRI/.

Accurate uncertainty estimation is crucial for deploying neural networks in risk-sensitive applications such as medical diagnosis. Monte Carlo Dropout is a widely used technique for approximating predictive uncertainty by performing stochastic forward passes with dropout during inference. However, using static dropout rates across all layers and inputs can lead to suboptimal uncertainty estimates, as it fails to adapt to the varying characteristics of individual inputs and network layers. Existing approaches optimize dropout rates during training using labeled data, resulting in fixed inference-time parameters that cannot adjust to new data distributions, compromising uncertainty estimates in Monte Carlo simulations.In this paper, we propose Rate-In, an algorithm that dynamically adjusts dropout rates during inference by quantifying the information loss induced by dropout in each layer’s feature maps. By treating dropout as controlled noise injection and leveraging information-theoretic principles, Rate-In adapts dropout rates per layer and per input instance without requiring ground truth labels. By quantifying the functional information loss in feature maps, we adaptively tune dropout rates to maintain perceptual quality across diverse medical imaging tasks and architectural configurations. Our extensive empirical study on synthetic data and real-world medical imaging tasks demonstrates that Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates without compromising predictive performance. Rate-In offers a practical, unsupervised, inference-time approach to optimizing dropout for more reliable predictive uncertainty estimation in critical applications.

We propose a model-agnostic, progressive test-time energy adaptation approach for medical image segmentation. Maintaining model performance across diverse medical datasets is challenging, as distribution shifts arise from inconsistent imaging protocols and patient variations. Unlike domain adaptation methods that require multiple passes through target data - impractical in clinical settings - our approach adapts pretrained models progressively as they process test data. Our method leverages a shape energy model trained on source data, which assigns an energy score at the patch level to segmentation maps: low energy represents in-distribution (accurate) shapes, while high energy signals out-of-distribution (erroneous) predictions. By minimizing this energy score at test time, we refine the segmentation model to align with the target distribution. To validate the effectiveness and adaptability, we evaluated our framework on eight public MRI (bSSFP, T1- and T2-weighted) and X-ray datasets spanning cardiac, spinal cord, and lung segmentation. We consistently outperform baselines both quantitatively and qualitatively.

Fine-tuning pre-trained vision-language models, like CLIP, has yielded success on diverse downstream tasks. However, several pain points persist for this paradigm: (i) directly tuning entire pre-trained models becomes both time-intensive and computationally costly. Additionally, these tuned models tend to become highly specialized, limiting their practicality for real-world deployment; (ii) recent studies indicate that pre-trained vision-language classifiers may overly depend on spurious features - patterns that correlate with the target in training data, but are not related to the true labeling function; and (iii) existing studies on mitigating the reliance on spurious features, largely based on the assumption that we can identify such features, does not provide definitive assurance for real-world applications. As a piloting study, this work focuses on exploring mitigating the reliance on spurious features for CLIP without using any group annotation. To this end, we systematically study the existence of spurious correlation on CLIP and CLIP+ERM. We first, following recent work on Deep Feature Reweighting (DFR), verify that last-layer retraining can greatly improve group robustness on pretrained CLIP. In view of them, we advocate a lightweight representation calibration method for fine-tuning CLIP, by first generating a calibration set using the pretrained CLIP, and then calibrating representations of samples within this set through contrastive learning, all without the need for group labels. Extensive experiments and in-depth visualizations on several benchmarks validate the effectiveness of our proposals, largely reducing reliance and significantly boosting the model generalization. Our codes will be available in here.

Recent studies on contrastive learning have achieved remarkable performance solely by leveraging few labels in the context of medical image segmentation. Existing methods mainly focus on instance discrimination and invariant mapping (i.e., pulling positive samples closer and negative samples apart in the feature space). However, they face three common pitfalls: (1) tailness: medical image data usually follows an implicit long-tail class distribution. Blindly leveraging all pixels in training hence can lead to the data imbalance issues, and cause deteriorated performance; (2) consistency: it remains unclear whether a segmentation model has learned meaningful and yet consistent anatomical features due to the intra-class variations between different anatomical features; and (3) diversity: the intra-slice correlations within the entire dataset have received significantly less attention. This motivates us to seek a principled approach for strategically making use of the dataset itself to discover similar yet distinct samples from different anatomical views. In this paper, we introduce a novel semi-supervised 2D medical image segmentation framework termed Mine yOur owNAnatomy (MONA), and make three contributions. First, prior work argues that every pixel equally matters to the model training; we observe empirically that this alone is unlikely to define meaningful anatomical features, mainly due to lacking the supervision signal. We show two simple solutions towards learning invariances-through the use of stronger data augmentations and nearest neighbors. Second, we construct a set of objectives that encourage the model to be capable of decomposing medical images into a collection of anatomical features in an unsupervised manner. Lastly, we both empirically and theoretically, demonstrate the efficacy of our MONA on three benchmark datasets with different labeled settings, achieving new state-of-the-art under different labeled semi-supervised settings. MONA makes minimal assumptions on domain expertise, and hence constitutes a practical and versatile solution in medical image analysis. We provide the PyTorch-like pseudo-code in supplementary.

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, if sampled, can significantly improve the performance. In reality, however, these samples may come from similar anatomical features and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone to misclassification, both of which typically lead to model collapse. In this paper, we propose $\texttt{ARCO}$, a semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building $\texttt{ARCO}$ through the concept of variance-reduced estimation, and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal in variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-art semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.

Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical textures. In this work, we present CASTformer, a novel type of adversarial transformers, for 2D medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CASTformer dramatically outperforms previous state-of-the-art transformer-based approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in Dice over previous models. Further qualitative experiments provide a more detailed picture of the model’s inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CASTformer a strong starting point for downstream medical image analysis tasks.

Automated segmentation in medical image analysis is a challenging task that requires a large amount of manually labeled data. However, most existing learning-based approaches usually suffer from limited manually annotated medical data, which poses a major practical problem for accurate and robust medical image segmentation. In addition, most existing semi-supervised approaches are usually not robust compared with the supervised counterparts, and also lack explicit modeling of geometric structure and semantic information, both of which limit the segmentation accuracy. In this work, we present SimCVD, a simple contrastive distillation framework that significantly advances state-of-the-art voxel-wise representation learning. We first describe an unsupervised training strategy, which takes two views of an input volume and predicts their signed distance maps of object boundaries in a contrastive objective, with only two independent dropout as mask. This simple approach works surprisingly well, performing on the same level as previous fully supervised methods with much less labeled data. We hypothesize that dropout can be viewed as a minimal form of data augmentation and makes the network robust to representation collapse. Then, we propose to perform structural distillation by distilling pair-wise similarities. We evaluate SimCVD on two popular datasets: the Left Atrial Segmentation Challenge (LA) and the NIH pancreas CT dataset. The results on the LA dataset demonstrate that, in two types of labeled ratios (i.e., 20% and 10%), SimCVD achieves an average Dice score of 90.85% and 89.03% respectively, a 0.91% and 2.22% improvement compared to previous best results. Our method can be trained in an end-to-end fashion, showing the promise of utilizing SimCVD as a general framework for downstream tasks, such as medical image synthesis, enhancement, and registration.

Reliable motion estimation and strain analysis using 3D+ time echocardiography (4DE) for localization and characterization of myocardial injury is valuable for early detection and targeted interventions. However, motion estimation is difficult due to the low-SNR that stems from the inherent image properties of 4DE, and intelligent regularization is critical for producing reliable motion estimates. In this work, we incorporated the notion of domain adaptation into a supervised neural network regularization framework. We first propose a semi-supervised Multi-Layered Perceptron (MLP) network with biomechanical constraints for learning a latent representation that is shown to have more physiologically plausible displacements. We extended this framework to include a supervised loss term on synthetic data and showed the effects of biomechanical constraints on the network's ability for domain adaptation. We validated the semi-supervised regularization method on in vivo data with implanted sonomicrometers. Finally, we showed the ability of our semi-supervised learning regularization approach to identify infarct regions using estimated regional strain maps with good agreement to manually traced infarct regions from postmortem excised hearts.

Deep learning models have shown their advantage in many different tasks, including neuroimage analysis. However, to effectively train a high-quality deep learning model, the aggregation of a significant amount of patient information is required. The time and cost for acquisition and annotation in assembling, for example, large fMRI datasets make it difficult to acquire large numbers at a single site. However, due to the need to protect the privacy of patient data, it is hard to assemble a central database from multiple institutions. Federated learning allows for population-level models to be trained without centralizing entities' data by transmitting the global model to local entities, training the model locally, and then averaging the gradients or weights in the global model. However, some studies suggest that private information can be recovered from the model gradients or weights. In this work, we address the problem of multi-site fMRI classification with a privacy-preserving strategy. To solve the problem, we propose a federated learning approach, where a decentralized iterative optimization algorithm is implemented and shared local model weights are altered by a randomization mechanism. Considering the systemic differences of fMRI distributions from different sites, we further propose two domain adaptation methods in this federated learning formulation. We investigate various practical aspects of federated model optimization and compare federated learning with alternative training strategies. Overall, our results demonstrate that it is promising to utilize multi-site data without data sharing to boost neuroimage analysis performance and find reliable disease-related biomarkers. Our proposed pipeline can be generalized to other privacy-sensitive medical data analysis problems. Our code is publicly available at: https://github.com/xxlya/Fed_ABIDE/.

The accurate segmentation of the brain surface in post-surgical computed tomography (CT) images is critical for image-guided neurosurgical procedures in epilepsy patients. Following surgical implantation of intracranial electrodes, surgeons require accurate registration of the post-implantation CT images to the pre-implantation functional and structural magnetic resonance imaging to guide surgical resection of epileptic tissue. One way to perform the registration is via surface matching. The key challenge in this setup is the CT segmentation, where the extraction of the cortical surface is difficult due to the missing parts of the skull and artifacts introduced from the electrodes. In this paper, we present a dictionary learning-based method to segment the brain surface in post-surgical CT images of epilepsy patients following surgical implantation of electrodes. We propose learning a model of locally oriented appearance that captures both the normal tissue and the artifacts found along this brain surface boundary. Utilizing a database of clinical epilepsy imaging data to train and test our approach, we demonstrate that our method using locally oriented image appearance both more accurately extracts the brain surface and better localizes electrodes on the post-operative brain surface compared to standard, non-oriented appearance modeling. In addition, we compare our method to a standard atlas-based segmentation approach and to a U-Net-based deep convolutional neural network segmentation method.

In the above-named document [ibid, vol. 33, no. 6, pp. 1275–1289, Jun. 2014], the funding information should have appeared as follows: "This work was supported in part by the National Institutes of Health (The National Heart, Lung, and Blood Institute) through these awards: R01HL082640, 5T32HL098069, and R01HL121226. The work of B. A. Lin was supported by an ASE Foundation Career Development Award."

Accurate and robust image registration is a fundamental task in medical image analysis applications, and requires non-rigid transformations with a large number of degrees of freedom. Statistical deformation models (SDMs) attempt to learn the distribution of non-rigid deformations, and can be used both to reduce the transformation dimensionality and to constrain the registration process. However, high-dimensional SDMs are difficult to train given orders of magnitude fewer training samples. In this paper, we utilize both a small set of annotated imaging data and a large set of unlabeled data to effectively learn an SDM of non-rigid transformations in a semi-supervised training (SST) framework. We demonstrate results applying this framework towards inter-subject registration of skull-stripped, magnetic resonance (MR) brain images. Our approach makes use of 39 labeled MR datasets to create a set of supervised registrations, which we augment with a set of over 1200 unsupervised registrations using unlabeled MRIs. Through leave-one-out cross validation, we show that SST of a non-rigid SDM results in a robust registration algorithm with significantly improved accuracy compared to standard, intensity-based registration, and does so with a 99% reduction in transformation dimensionality.

Quantitative analysis of left ventricular deformation can provide valuable information about the extent of disease as well as the efficacy of treatment. In this work, we develop an adaptive multi-level compactly supported radial basis approach for deformation analysis in 3D+time echocardiography. Our method combines displacement information from shape tracking of myocardial boundaries (derived from B-mode data) with mid-wall displacements from radio-frequency-based ultrasound speckle tracking. We evaluate our methods on open-chest canines (N=8) and show that our combined approach is better correlated to magnetic resonance tagging-derived strains than either individual method. We also are able to identify regions of myocardial infarction (confirmed by postmortem analysis) using radial strain values obtained with our approach.

During neurosurgery, nonrigid brain deformation may affect the reliability of tissue localization based on preoperative images. To provide accurate surgical guidance in these cases, preoperative images must be updated to reflect the intraoperative brain. This can be accomplished by warping these preoperative images using a biomechanical model. Due to the possible complexity of this deformation, intraoperative information is often required to guide the model solution. In this paper, a linear elastic model of the brain is developed to infer volumetric brain deformation associated with measured intraoperative cortical surface displacement. The developed model relies on known material properties of brain tissue, and does not require further knowledge about intraoperative conditions. To provide an initial estimation of volumetric model accuracy, as well as determine the model's sensitivity to the specified material parameters and surface displacements, a realistic brain phantom was developed. Phantom results indicate that the linear elastic model significantly reduced localization error due to brain shift, from > 16 mm to under 5 mm, on average. In addition, though in vivo quantitative validation is necessary, preliminary application of this approach to images acquired during neocortical epilepsy cases confirms the feasibility of applying the developed model to in vivo data.

Fiber tracking in diffusion tensor magnetic resonance images (DTIs) reveals 3-D structural connectivity of the brain conveniently and thus is a viable tool for investigating neural differences. Unfortunately, local noise, image artifacts and numerical tracking errors during integration-based techniques are cumulative. Prematurely terminated fibers and under-sampled fiber bundles result in incomplete reconstruction of white matter fiber tracts and hence incorrect anatomical measurements. Quantitative cross-subject tract analysis, which is critical for abnormality detection, is complicated by inefficient and inaccurate tract reconstruction and normalization from fiber bundles. Because of the above problems, we propose a parcellation method that aims for lower sensitivity to initialization and local orientation error by directly segmenting full white matter tracts (Fasciculography), rather than reconstructing individual curves, from diffusion tensor fields. A fast, robust volumetric, and intrinsically normalized solution is achieved by noise-filtering using a generic parametrized tract model to prevent premature tract termination. At the same time, orientation information reduces the search space, significantly speeding up the tract parcellation process with less human intervention. Detailed comparisons against streamline tracking, shortest-path tracking, and nonrigid registration using synthetic and real DTIs confirmed the superior properties of Fasciculography. Since a normalized tract can be delineated interactively in a just few seconds using the proposed method, accurate high volume tract comparisons become feasible.

External beam radiation therapy (EBRT) for the treatment of cancer enables accurate placement of radiation dose on the cancerous region. However, the deformation of soft tissue during the course of treatment, such as in cervical cancer, presents significant challenges for the delineation of the target volume and other structures of interest. Furthermore, the presence and regression of pathologies such as tumors may violate registration constraints and cause registration errors. In this paper, automatic segmentation, nonrigid registration and tumor detection in cervical magnetic resonance (MR) data are addressed simultaneously using a unified Bayesian framework. The proposed novel method can generate a tumor probability map while progressively identifying the boundary of an organ of interest based on the achieved nonrigid transformation. The method is able to handle the challenges of significant tumor regression and its effect on surrounding tissues. The new method was compared to various currently existing algorithms on a set of 36 MR data from six patients, each patient has six T2-weighted MR cervical images. The results show that the proposed approach achieves an accuracy comparable to manual segmentation and it significantly outperforms the existing registration algorithms. In addition, the tumor detection result generated by the proposed method has a high agreement with manual delineation by a qualified clinician.

During neurosurgery, nonrigid brain deformation prevents preoperatively-acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track intraoperative cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration from the surface deformation estimation, a framework that can solve for disparate and often competing variables is needed. Game theory, which was developed to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, game theory is applied to cortical surface tracking during neocortical epilepsy surgery and used to infer information about the physical processes of brain surface deformation and image acquisition. The method is successfully applied to eight in vivo cases, resulting in an 81% decrease in mean surface displacement error. This includes a case in which some of the initial camera calibration parameters had errors of 70%. Additionally, the advantages of using a game theoretic approach in neocortical epilepsy surgery are clearly demonstrated in its robustness to initial conditions.

Shape registration is often involved in computing statistical differences between groups of shapes, which is a key aspect of morphometric study. The results of shape difference are found to be sensitive to registration, i.e., different registration methods lead to varied results. This raises the question of how to improve the reliability of registration procedures. This paper proposes a perturbation scheme, which perturbs registrations by feeding them with different resampled shape groups, and then aggregates the resulting shape differences. Experiments are conducted using three typical registration algorithms on both synthetic and biomedical shapes, where more reliable inter-group shape differences are found under the proposed scheme.

Computing the morphological similarity of diffusion tensors (DTs) at neighboring voxels within a DT image, or at corresponding locations across different DT images, is a fundamental and ubiquitous operation in the postprocessing of DT images. The morphological similarity of DTs typically has been computed using either the principal directions (PDs) of DTs (i.e., the direction along which water molecules diffuse preferentially) or their tensor elements. Although comparing PDs allows the similarity of one morphological feature of DTs to be visualized directly in eigenspace, this method takes into account only a single eigenvector, and it is therefore sensitive to the presence of noise in the images that can introduce error intothe estimation of that vector. Although comparing tensor elements, rather than PDs, is comparatively more robust to the effects of noise, the individual elements of a given tensor do not directly reflect the diffusion properties of water molecules. We propose a measure for computing the morphological similarity of DTs that uses both their eigenvalues and eigenvectors, and that also accounts for the noise levels present in DT images. Our measure presupposes that DTs in a homogeneous region within or across DT images are random perturbations of one another in the presence of noise. The similarity values that are computed using our method are smooth (in the sense that small changes in eigenvalues and eigenvectors cause only small changes in similarity), and they are symmetric when differences in eigenvalues and eigenvectors are also symmetric. In addition, our method does not presuppose that the corresponding eigenvectors across two DTs have been identified accurately, an assumption that is problematic in the presence of noise. Because we compute the similarity between DTs using their eigenspace components, our similarity measure relates directly to both the magnitude and the direction of the diffusion of water molecules. The favorable performance characteristics of our measure offer the prospect of substantially improving additional postprocessing operations that are commonly performed on DTI datasets, such as image segmentation, fiber tracking, noise filtering, and spatial normalization.

Neurosurgical planning and image guided neurosurgery require the visualization of multimodal data obtained from various functional and structural image modalities, such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), functional MRI, Single photon emission computed tomography (SPECT) and so on. In the case of epilepsy neurosurgery for example, these images are used to identify brain regions to guide intracranial electrode implantation and resection. Generally, such data is visualized using 2D slices and in some cases using a 3D volume rendering along with the functional imaging results. Visualizing the activation region effectively by still preserving sufficient surrounding brain regions for context is exceedingly important to neurologists and surgeons. We present novel interaction techniques for visualization of multimodal data to facilitate improved exploration and planning for neurosurgery. We extended the line widget from VTK to allow surgeons to control the shape of the region of the brain that they can visually crop away during exploration and surgery. We allow simple spherical, cubical, ellipsoidal and cylindrical (probe aligned cuts) for exploration purposes. In addition we integrate the cropping tool with the image-guided navigation system used for epilepsy neurosurgery. We are currently investigating the use of these new tools in surgical planning and based on further feedback from our neurosurgeons we will integrate them into the setup used for image-guided neurosurgery.

During neurosurgery, nonrigid brain deformation prevents preoperatively acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration and surface deformation, a framework is needed which can solve for disparate and often competing variables. Game theory, which was developed specifically to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, we apply game theory to cortical surface tracking and use it to infer information about the physical processes of brain deformation and image acquisition.

Interest in the morphometric analysis of the brain and its subregions has recently intensified because growth or degeneration of the brain in health or illness affects not only the volume but also the shape of cortical and subcortical brain regions, and new image processing techniques permit detection of small and highly localized perturbations in shape or localized volume, with remarkable precision. An appropriate statistical representation of the shape of a brain region is essential, however, for detecting, localizing, and interpreting variability in its surface contour and for identifying differences in volume of the underlying tissue that produce that variability across individuals and groups of individuals. Our statistical representation of the shape of a brain region is defined by a reference region for that region and by a Gaussian random field (GRF) that is defined across the entire surface of the region. We first select a reference region from a set of segmented brain images of healthy individuals. The GRF is then estimated as the signed Euclidean distances between points on the surface of the reference region and the corresponding points on the corresponding region in images of brains that have been coregistered to the reference. Correspondences between points on these surfaces are defined through deformations of each region of a brain into the coordinate space of the reference region using the principles of fluid dynamics. The warped, coregistered region of each subject is then unwarped into its native space, simultaneously bringing into that space the map of corresponding points that was established when the surfaces of the subject and reference regions were tightly coregistered. The proposed statistical description of the shape of surface contours makes no assumptions, other than smoothness, about the shape of the region or its GRF. The description also allows for the detection and localization of statistically significant differences in the shapes of the surfaces across groups of subjects at both a fine and coarse scale. We demonstrate the effectiveness of these statistical methods by applying them to study differences in shape of the amygdala and hippocampus in a large sample of normal subjects and in subjects with attention deficit/hyperactivity disorder (ADHD).

A novel method for the segmentation of multiple objects from three-dimensional (3-D) medical images using interobject constraints is presented. Our method is motivated by the observation that neighboring structures have consistent locations and shapes that provide configurations and context that aid in segmentation. We define a maximum a posteriori (MAP) estimation framework using the constraining information provided by neighboring objects to segment several objects simultaneously. We introduce a representation for the joint density function of the neighbor objects, and define joint probability distributions over the variations of the neighboring shape and position relationships of a set of training images. In order to estimate the MAP shapes of the objects, we formulate the model in terms of level set functions, and compute the associated Euler-Lagrange equations. The contours evolve both according to the neighbor prior information and the image gray level information. This method is useful in situations where there is limited interobject information as opposed to robust global atlases. In addition, we compare our level set representation of the object shape to the point distribution model. Results and validation from experiments on synthetic data and medical imagery in two-dimensional and 3-D are demonstrated.

For patient setup verification in external beam radiotherapy (EBRT) of prostate cancer, we developed an information theoretic registration framework, called the minimax entropy registration framework, to simultaneously and iteratively segment portal images and register them to three-dimensional (3-D) computed tomography (CT) image data. The registration framework has two steps, the max step and the min step, and evaluates appropriate entropies to estimate segmentations of the portal images and to find the transformation parameters. In the initial version of the algorithm (Bansal et al. 1999), we assumed image pixels to be independently distributed, an assumption not true in general. Thus, to better segment the portal images and to improve the accuracy of the estimated registration parameters, in this initial formulation of the problem, the correlation among pixel intensities is modeled using a one-dimensional Markov random process. Line processes are incorporated into the model to improve the estimation of segmentation of the portal images. In the max step, the principle of maximum entropy is invoked to estimate the probability distribution on the segmentations. The estimated distribution is then incorporated into the min step to estimate the registration parameters. Performance of the proposed framework is evaluated and compared to that of a mutual information-based registration algorithm using both simulated and real patient data. In the proposed registration framework, registration of the 3-D CT image and the portal images is guided by an estimated segmentation of the pelvic bone. However, as the prostate can move with respect to the pelvic structure, further localization of the prostate using ultrasound image data is required, an issue to be further explored in future.

Medical image analysis has grown and evolved tremendously in the last 30 years. The distinctive nature of the problems encountered have led to the development of a significant body of work addressing such issues as fully three-dimensional data,

This paper describes a new method for determining correspondence between points on pairs of surfaces based on shape using a combination of geodesic distance and surface curvature. An initial sparse set of corresponding points is generated using a shape-based matching procedure. Geodesic interpolation is employed in order to capture the complex surface. In addition, surface correspondence and triangulation are computed simultaneously in a hierarchical way. Results applied to human cerebral cortical surfaces are shown to evaluate the approach.

OBJECTIVE: Currently, when cytopathology images are archived, they are typically stored with a limited text-based description of their content. Such a description inherently fails to quantify the properties of an image and refers to an extremely small fraction of its information content. This paper describes a method for automatically indexing images of individual cells and their associated diagnoses by computationally derived cell descriptors. This methodology may serve to better index data contained in digital image databases, thereby enabling cytologists and pathologists to cross-reference cells of unknown etiology or nature. DESIGN: The indexing method, implemented in a program called PathMaster, uses a series of computer-based feature extraction routines. Descriptors of individual cell characteristics generated by these routines are employed as indexes of cell morphology, texture, color, and spatial orientation. MEASUREMENTS: The indexing fidelity of the program was tested after populating its database with images of 152 lymphocytes/lymphoma cells captured from lymph node touch preparations stained with hematoxylin and eosin. Images of "unknown" lymphoid cells, previously unprocessed, were then submitted for feature extraction and diagnostic cross-referencing analysis. RESULTS: PathMaster listed the correct diagnosis as its first differential in 94 percent of recognition trials. In the remaining 6 percent of trials, PathMaster listed the correct diagnosis within the first three "differentials." CONCLUSION: PathMaster is a pilot cell image indexing program/search engine that creates an indexed reference of images. Use of such a reference may provide assistance in the diagnostic/prognostic process by furnishing a prioritized list of possible identifications for a cell of uncertain etiology.

This paper describes two new atlas-based methods of 2D single modality non-rigid registration using the combined power of physical and statistical shape models. The transformations are constrained to be consistent with the physical properties of deformable elastic solids in the first method and those of viscous fluids in the second, to maintain smoothness and continuity. A Bayesian formulation, based on each physical model, an intensity similarity measure, and statistical shape information embedded in corresponding boundary points, is employed to derive more accurate and robust approaches to non-rigid registration. A dense set of forces arises from the intensity similarity measure to accommodate complex anatomical details. A sparse set of forces constrains consistency with statistical shape models derived from a training set. A number of experiments were performed on both synthetic and real medical images of the brain and heart to evaluate the approaches. It is shown that statistical boundary shape information significantly augments and improves physical model-based non-rigid registration and the two methods we present each have advantages under different conditions.

We propose a unified framework for boundary finding, where a Bayesian formulation, based on prior knowledge and the edge information of the input image (likelihood), is employed. The prior knowledge in our framework is based on principal component analysis of four different covariance matrices corresponding to independence, smoothness, statistical shape, and combined models, respectively. Indeed, snakes, modal analysis, Fourier descriptors, and point distribution models can be derived from or linked to our approaches of different prior models. When the true training set does not contain enough variability to express the full range of deformations, a mixed covariance matrix uses a combined prior of the smoothness and statistical variation modes. It adapts gradually to use more statistical modes of variation as larger data sets are available.

The cortex is the outermost thin layer of gray matter in the brain; geometric measurement of the cortex helps in understanding brain anatomy and function. In the quantitative analysis of the cortex from MR images, extracting the structure and obtaining a representation for various measurements are key steps. While manual segmentation is tedious and labor intensive, automatic reliable efficient segmentation and measurement of the cortex remain challenging problems, due to its convoluted nature. Here we present a new approach of coupled-surfaces propagation, using level set methods to address such problems. Our method is motivated by the nearly constant thickness of the cortical mantle and takes this tight coupling as an important constraint. By evolving two embedded surfaces simultaneously, each driven by its own image-derived information while maintaining the coupling, a final representation of the cortical bounding surfaces and an automatic segmentation of the cortex are achieved. Characteristics of the cortex, such as cortical surface area, surface curvature, and cortical thickness, are then evaluated. The level set implementation of surface propagation offers the advantage of easy initialization, computational efficiency, and the ability to capture deep sulcal folds. Results and validation from various experiments on both simulated and real three-dimensional (3-D) MR images are provided.

We propose an approach for boundary finding where the correspondence of a subset of boundary points to a model is simultaneously determined. Global shape parameters derived from the statistical variation of object boundary points in a training set are used to model the object. A Bayesian formulation, based on this prior knowledge and the edge information of the input image, is employed to find the object boundary with its subset points in correspondence with boundaries in the training set or the mean boundary. We compared the use of a generic smoothness prior and a uniform independent prior with the training set prior in order to demonstrate the power of this statistical information. A number of experiments were performed on both synthetic and real medical images of the brain and heart to evaluate the approach, including the validation of the dependence of the method on image quality, different initialization and prior information.

The problem of segmenting a volumetric layer of finite thickness is encountered in several important areas within medical image analysis. Key examples include the extraction of the cortical gray matter of the brain and the left ventricle myocardium of the heart. The coupling between the two bounding surfaces of such a layer provides important information that helps to solve the segmentation problem. Here we propose a new approach of coupled surfaces propagation via level set methods, which takes into account coupling as an important constraint. By evolving two embedded surfaces simultaneously, each driven by its own image-derived information while maintaining the coupling, we capture a representation of the two bounding surfaces and achieve automatic segmentation on the layer. Characteristic gray level values, instead of image gradient information alone, are incorporated in deriving the useful image information to drive the surface propagation, which enables our approach to capture the homogeneity inside the layer. The level set implementation offers the advantage of easy initialization, computational efficiency and the ability to capture deep folds of the sulci. As a test example, we apply our approach to unedited 3D Magnetic Resonance (MR) brain images. Our algorithm automatically isolates the brain from non-brain structures and recovers the cortical gray matter.

This paper describes two new atlas-based methods of 2D single modality non-rigid registration using the combined power of physical and statistical shape models. The transformations are constrained to be consistent with the physical properties of deformable elastic solids in the first method and those of viscous fluids in the second to maintain smoothness and continuity. A Bayesian formulation, based on each physical model, on an intensity similarity measure, and on statistical shape information embedded in corresponding boundary points, is employed to derive more accurate and robust approaches to non-rigid registration. A dense set of forces arises from the intensity similarity measure to accommodate complex anatomical details. A sparse set of forces constrains consistency with statistical shape models derived from a training set. A number of experiments were performed on both synthetic and real medical images of the brain and heart to evaluate the approaches. It is shown that statistical boundary shape information significantly augments and improves physical model based non-rigid registration and the two methods we present each have advantages under different conditions.

The wide availability of high resolution magnetic resonance images (MRI) of the brain has facilitated tremendous progress in neuroscience. Accurate automated segmentation and quantification of neuroanatomical structure from such images is crucial for the advancement of the understanding of brain morphology, both in normal variation and in disease. Gradient-based deformable surface finding is a powerful technique for locating structure in three-dimensional images. However, it often suffers from poorly defined edges and noise. This paper proposes a gradient-based deformable surface finding approach that integrates region information. This makes the resulting procedure more robust to noise and improper initialization. In addition, prior shape information may be incorporated. The algorithm uses Gauss's Divergence theorem to find the surface of a homogeneous region-classified area in the image and integrates this with a gray-level gradient-based surface finder. Experimental results on synthetic and MR brain images show that a significant improvement is achieved as a consequence of the use of this extra information. Further, these improvements are achieved with little increase in computational overhead, an advantage derived from the application of Gauss's Divergence theorem.

Segmentation of (noisy) images containing a complex ensemble of objects is difficult to achieve on the basis of local image information only. It is advantageous to attack the problem of object boundary extraction by a model-based segmentation procedure. Segmentation is achieved by tuning the parameters of the geometrical model in such a way that the boundary template locates and describes the object in the image in an optimal way. The optimality of the solution is based on an objective function taking into account image information as well as the shape of the template. Objective functions in literature are mainly based on the gradient magnitude and a measure describing the smoothness of the template. In this contribution, we propose a new image objective function based on directional gradient information derived from Gaussian smoothed derivatives of the image data. The proposed method is designed to accurately locate an object boundary even in the case of a conflicting object positioned close to the object of interest. We further introduce a new smoothness objective to ensure the physical feasibility of the contour. The method is evaluated on artificial data. Results on real medical images show that the method is very effective in accurately locating object boundaries in very complex images.

Accurately segmenting and quantifying structures is a key issue in biomedical image analysis. The two conventional methods of image segmentation, region-based segmentation, and boundary finding, often suffer from a variety of limitations. Here the authors propose a method which endeavors to integrate the two approaches in an effort to form a unified approach that is robust to noise and poor initialization. The authors' approach uses Green's theorem to derive the boundary of a homogeneous region-classified area in the image and integrates this with a gray level gradient-based boundary finder. This combines the perceptual notions of edge/shape information with gray level homogeneity. A number of experiments were performed both on synthetic and real medical images of the brain and heart to evaluate the new approach, and it is shown that the integrated method typically performs better when compared to conventional gradient-based deformable boundary finding. Further, this method yields these improvements with little increase in computational overhead, an advantage derived from the application of the Green's theorem.

Describes a new global shape parameterization for smoothly deformable three-dimensional (3-D) objects, such as those found in biomedical images, whose diversity and irregularity make them difficult to represent in terms of fixed features or parts. This representation is used for geometric surface matching to 3-D medical image data, such as from magnetic resonance imaging (MRI). The parameterization decomposes the surface into sinusoidal basis functions. Four types of surfaces are modeled: tori, open surfaces, closed surfaces and tubes. This parameterization allows a wide variety of smooth surfaces to be described with a small number of parameters. Extrinsic model-based information is incorporated by introducing prior probabilities on the parameters. Surface finding is formulated as an optimization problem. Results of the method applied to synthetic images and 3-D medical images of the heart and brain are presented.

Cardiac SPECT (single photon emission computed tomography) is an important tool for evaluating heart disease in terms of diagnosis and treatment, especially for determining myocardial perfusion and thus the degree of myocardial injury. The distortion of left ventricular (LV) geometry can cause errors in defect size determination from SPECT perfusion images due to partial volume effects. Constrained iterative restoration, using an anatomic constraint from a registered magnetic resonance (MR) image of the heart, can be used to correct these errors and thus improve the interpretation and measurement of cardiac perfusion images.

Accurately segmenting and quantifying structures is a key issue in biomedical image analysis. The two conventional methods of image segmentation, region-based segmentation and boundary finding, often suffer from a variety of limitations. We propose a method which endeavors to integrate the two approaches in an effort to form a unified approach that is robust to noise and poor initialization. Our approach uses Green's theorem to derive the boundary of a homogeneous region-classified area in the image and integrates this with a grey-level-gradient-based boundary finder. This combines the perceptual notions of edge/shape information with gray level homogeneity.>

Segmentation using boundary finding is enhanced both by considering the boundary as a whole and by using model-based global shape information. The authors apply flexible constraints, in the form of a probabilistic deformable model, to the problem of segmenting natural 2-D objects whose diversity and irregularity of shape make them poorly represented in terms of fixed features or form. The parametric model is based on the elliptic Fourier decomposition of the boundary. Probability distributions on the parameters of the representation bias the model to a particular overall shape while allowing for deformations. Boundary finding is formulated as an optimization problem using a maximum a posteriori objective function. Results of the method applied to real and synthetic images are presented, including an evaluation of the dependence of the method on prior information and image quality.>

The problem of measuring the motion of deformable objects from image sequences is addressed. The approach is based upon modeling the overall boundary of the object as a deformable contour and then tracking local segments of the contour through the temporal sequence. Motion computation involves first matching the local segments between pairs of contours by minimizing the deformation between the segments using a measure of bending energy. Results from the match process are incorporated into an optimization functional, along with a general smoothness term, whose local minimum results in a smooth flow field that is consistent with the match data. The computation is performed for all pairs of frames in the temporal sequence, resulting in a composite flow field over the entire sequence. The technique is applied to synthetic contour sequences and the problem of tracking left ventricular (LV) endocardial motion from medical image sequences.>

Segmentation using boundary finding is enhanced both by considering the boundary as a whole and by using model-based shape information. Flexible constraints, in the form of a probabilistic deformable model, are applied to the problem of segmenting natural objects whose diversity and irregularity of shape makes them poorly represented in terms of fixed features of forms. The parametric model is based on the elliptic Fourier decomposition of the boundary. The segmentation problem is solved as an optimization problem, where the best match between the boundary (as defined by the parameter vector) and the image data is found. Initial experimentation shows good results on a variety of images.>