Functional magnetic resonance imaging (fMRI) is a powerful tool for investigating human brain function. However, the high cost of data acquisition and the inherent subjectivity of psychiatric rating scales often lead to datasets with small sample sizes and variable label quality, especially when targeting a specific neurological condition. Combined with the inherently high dimensionality of fMRI data, these limitations substantially increase the risk of model overfitting. Recent years have seen growing interest in developing fMRI foundation models by combining multiple datasets; however, the computational resources needed for pretraining and fine-tuning are often prohibitive. We show that a lightweight self-supervised framework yields representations that generalize across diverse downstream tasks, outperforming fully supervised baselines and approaching the performance of large-scale models. We introduce BrainSimSiam, a data-efficient self-supervised representation learning framework that leverages positive-only data pairs to learn robust and generalizable features. We demonstrate that the learned representations achieve strong performance across multiple downstream classification and regression tasks, highlighting the potential of BrainSimSiam for data-limited neuroimaging applications. Our implementation is available in https://github.com/Jiyao96/BrainSimSiam-fMRI/.

Head movement poses a significant challenge in brain positron emission tomography (PET) imaging, resulting in image artifacts and tracer uptake quantification inaccuracies. Effective head motion estimation and correction are crucial for precise quantitative image analysis and accurate diagnosis of neurological disorders. Hardware-based motion tracking (HMT) has limited applicability in real-world clinical practice. To overcome this limitation, we propose a deep-learning head motion correction approach with cross-attention (DL-HMC++) to predict rigid head motion from one-second 3D PET raw data. DL-HMC++ is trained in a supervised manner by leveraging existing dynamic PET scans with gold-standard motion measurements from external HMT. We evaluate DL-HMC++ on two PET scanners (HRRT and mCT) and four radiotracers (18F-FDG,18F-FPEB,11C-UCB-J, and11C-LSN3172176) to demonstrate the effectiveness and generalization of the approach in large cohort PET studies. Quantitative and qualitative results demonstrate that DL-HMC++ consistently outperforms state-of-the-art data-driven motion estimation methods, producing motion-free images with clear delineation of brain structures and reduced motion artifacts that are indistinguishable from gold-standard HMT. Brain region of interest standard uptake value analysis exhibits average difference ratios between DL-HMC++ and gold-standard HMT to be 1.2±0.5% for HRRT and 0.5±0.2% for mCT. DL-HMC++ demonstrates the potential for data-driven PET head motion correction to remove the burden of HMT, making motion correction accessible to clinical populations beyond research settings. The code is available at https://github.com/maxxxxxxcai/DL-HMC-TMI.

We propose a method for test-time adaptation of pretrained depth completion models. Depth completion models, trained on some ``source'' data, often predict erroneous outputs when transferred to ``target'' data captured in novel environmental conditions due to a covariate shift. The crux of our method lies in quantifying the likelihood of depth predictions belonging to the source data distribution. The challenge is in the lack of access to out-of-distribution (target) data prior to deployment. Hence, rather than making assumptions regarding the target distribution, we utilize adversarial perturbations as a mechanism to explore the data space. This enables us to train an energy model that scores local regions of depth predictions as in- or out-of-distribution. We update the parameters of pretrained depth completion models at test time to minimize energy, effectively aligning test-time predictions to those of the source distribution. We call our method ``Energy-based Test-time Adaptation'', or ETA for short. We evaluate our method across three indoor and three outdoor datasets, where ETA improve over the previous state-of-the-art method by an average of 6.94% for outdoors and 10.23% for indoors. Project Page: https://fuzzythecat.github.io/eta.

We propose a model-agnostic, progressive test-time energy adaptation approach for medical image segmentation. Maintaining model performance across diverse medical datasets is challenging, as distribution shifts arise from inconsistent imaging protocols and patient variations. Unlike domain adaptation methods that require multiple passes through target data - impractical in clinical settings - our approach adapts pretrained models progressively as they process test data. Our method leverages a shape energy model trained on source data, which assigns an energy score at the patch level to segmentation maps: low energy represents in-distribution (accurate) shapes, while high energy signals out-of-distribution (erroneous) predictions. By minimizing this energy score at test time, we refine the segmentation model to align with the target distribution. To validate the effectiveness and adaptability, we evaluated our framework on eight public MRI (bSSFP, T1- and T2-weighted) and X-ray datasets spanning cardiac, spinal cord, and lung segmentation. We consistently outperform baselines both quantitatively and qualitatively.

Magnetic Resonance Spectroscopic Imaging (MRSI) is a non-invasive imaging technique for studying metabolism and has become a crucial tool for understanding neurological diseases , cancers and diabetes. High spatial resolution MRSI is needed to characterize lesions, but in practice MRSI is acquired at low resolution due to time and sensitivity restrictions caused by the low metabolite concentrations. Therefore, there is an imperative need for a post-processing approach to generate high-resolution MRSI from low-resolution data that can be acquired fast and with high sensitivity. Deep learning-based super-resolution methods provided promising results for improving the spatial resolution of MRSI, but they still have limited capability to generate accurate and high-quality images. Recently, diffusion models have demonstrated superior learning capability than other generative models in various tasks, but sampling from diffusion models requires iterating through a large number of diffusion steps, which is time-consuming. This work introduces a Flow-based Truncated Denoising Diffusion Model (FTDDM) for super-resolution MRSI, which shortens the diffusion process by truncating the diffusion chain, and the truncated steps are estimated using a normalizing flow-based network. The network is conditioned on upscaling factors to enable multi-scale super-resolution. To train and evaluate the deep learning models, we developed a 1 H-MRSI dataset acquired from 25 high-grade glioma patients. We demonstrate that FTDDM outperforms existing generative models while speeding up the sampling process by over 9-fold compared to the baseline diffusion model. Neuroradiologists’ evaluations confirmed the clinical advantages of our method, which also supports uncertainty estimation and sharpness adjustment, extending its potential clinical applications.

Positron Emission Tomography (PET) is an important clinical imaging tool but inevitably introduces radiation exposure to patients and healthcare providers. Reducing the tracer injection dose and eliminating the CT acquisition for attenuation correction can reduce the overall radiation dose, but often results in PET with high noise and bias. Thus, it is desirable to develop 3D methods to translate the non-attenuation-corrected low-dose PET (NAC-LDPET) into attenuation-corrected standard-dose PET (AC-SDPET). Recently, diffusion models have emerged as a new state-of-the-art deep learning method for image-to-image translation, better than traditional CNN-based methods. However, due to the high computation cost and memory burden, it is largely limited to 2D applications. To address these challenges, we developed a novel 2.5D Multi-view Averaging Diffusion Model (MADM) for 3D image-to-image translation with application on NAC-LDPET to AC-SDPET translation. Specifically, MADM employs separate diffusion models for axial, coronal, and sagittal views, whose outputs are averaged in each sampling step to ensure the 3D generation quality from multiple views. To accelerate the 3D sampling process, we also proposed a strategy to use the CNN-based 3D generation as a prior for the diffusion model. Our experimental results on human patient studies suggested that MADM can generate high-quality 3D translation images, outperforming previous CNN-based and Diffusion-based baseline methods. The code is available at https://github.com/tianqic/MADM.

Low-dose PET offers a valuable means of minimizing radiation exposure in PET imaging. However, the prevalent practice of employing additional CT scans for generating attenuation maps ( -map) for PET attenuation correction significantly elevates radiation doses. To address this concern and further mitigate radiation exposure in low-dose PET exams, we propose an innovative Population-prior-aided Over-Under-Representation Network (POUR-Net) that aims for high-quality attenuation map generation from low-dose PET. First, POUR-Net incorporates an Over-Under-Representation Network (OUR-Net) to facilitate efficient feature extraction, encompassing both low-resolution abstracted and fine-detail features, for assisting deep generation on the full-resolution level. Second, complementing OUR-Net, a population prior generation machine (PPGM) utilizing a comprehensive CT-derived -map dataset, provides additional prior information to aid OUR-Net generation. The integration of OUR-Net and PPGM within a cascade framework enables iterative refinement of -map generation, resulting in the production of high-quality -maps. Experimental results underscore the effectiveness of POUR-Net, showing it as a promising solution for accurate CT-free low-count PET attenuation correction, which also surpasses the performance of previous baseline methods.

Fine-tuning pre-trained vision-language models, like CLIP, has yielded success on diverse downstream tasks. However, several pain points persist for this paradigm: (i) directly tuning entire pre-trained models becomes both time-intensive and computationally costly. Additionally, these tuned models tend to become highly specialized, limiting their practicality for real-world deployment; (ii) recent studies indicate that pre-trained vision-language classifiers may overly depend on spurious features - patterns that correlate with the target in training data, but are not related to the true labeling function; and (iii) existing studies on mitigating the reliance on spurious features, largely based on the assumption that we can identify such features, does not provide definitive assurance for real-world applications. As a piloting study, this work focuses on exploring mitigating the reliance on spurious features for CLIP without using any group annotation. To this end, we systematically study the existence of spurious correlation on CLIP and CLIP+ERM. We first, following recent work on Deep Feature Reweighting (DFR), verify that last-layer retraining can greatly improve group robustness on pretrained CLIP. In view of them, we advocate a lightweight representation calibration method for fine-tuning CLIP, by first generating a calibration set using the pretrained CLIP, and then calibrating representations of samples within this set through contrastive learning, all without the need for group labels. Extensive experiments and in-depth visualizations on several benchmarks validate the effectiveness of our proposals, largely reducing reliance and significantly boosting the model generalization. Our codes will be available in here.

Accurate cardiac segmentation in cross-modality images plays an important role in the quantitative analysis of the heart to diagnose cardiovascular diseases. However, achieving high performance in cross-modality segmentation is hindered by the time-consuming annotation and modality gap. While some approaches employ Unsupervised Domain Adaptation (UDA) through adversarial learning to address the issue, it still remains challenging due to the instability of the adversarial generative models. In this work, we propose Symmetric Consistency with Cross-Domain Mixup (SCCDM), integrated with the teacher-student model for cross-modality cardiac segmentation. Specifically, we introduce symmetric consistency across the domains for two mixed data to diversify the data distribution from both the source domain and target domain. Extensive experiments on a public cardiac dataset demonstrate that SCCDM achieves superior domain adaptation performance for cardiac segmentation compared to state-of-the-art methods.

Image registration is an essential step in many medical image analysis tasks. Traditional methods for image registration are primarily optimization-driven, finding the optimal deformations that maximize the similarity between two images. Recent learning-based methods, trained to directly predict transformations between two images, run much faster, but suffer from performance deficiencies due to domain shift. Here we present a new neural network based image registration framework, called NIR (Neural Image Registration), which is based on optimization but utilizes deep neural networks to model deformations between image pairs. NIR represents the transformation between two images with a continuous function implemented via neural fields, receiving a 3D coordinate as input and outputting the corresponding deformation vector. NIR provides two ways of generating deformation field: directly output a displacement vector field for general deformable registration, or output a velocity vector field and integrate the velocity field to derive the deformation field for diffeomorphic image registration. The optimal registration is discovered by updating the parameters of the neural field via stochastic mini-batch gradient descent. We describe several design choices that facilitate model optimization, including coordinate encoding, sinusoidal activation, coordinate sampling, and intensity sampling. NIR is evaluated on two 3D MR brain scan datasets, demonstrating highly competitive performance in terms of both registration accuracy and regularity. Compared to traditional optimization-based methods, our approach achieves better results in shorter computation times. In addition, our methods exhibit performance on a cross-dataset registration task, compared to the pre-trained learning-based methods.

Recent studies on contrastive learning have achieved remarkable performance solely by leveraging few labels in the context of medical image segmentation. Existing methods mainly focus on instance discrimination and invariant mapping (i.e., pulling positive samples closer and negative samples apart in the feature space). However, they face three common pitfalls: (1) tailness: medical image data usually follows an implicit long-tail class distribution. Blindly leveraging all pixels in training hence can lead to the data imbalance issues, and cause deteriorated performance; (2) consistency: it remains unclear whether a segmentation model has learned meaningful and yet consistent anatomical features due to the intra-class variations between different anatomical features; and (3) diversity: the intra-slice correlations within the entire dataset have received significantly less attention. This motivates us to seek a principled approach for strategically making use of the dataset itself to discover similar yet distinct samples from different anatomical views. In this paper, we introduce a novel semi-supervised 2D medical image segmentation framework termed Mine yOur owNAnatomy (MONA), and make three contributions. First, prior work argues that every pixel equally matters to the model training; we observe empirically that this alone is unlikely to define meaningful anatomical features, mainly due to lacking the supervision signal. We show two simple solutions towards learning invariances-through the use of stronger data augmentations and nearest neighbors. Second, we construct a set of objectives that encourage the model to be capable of decomposing medical images into a collection of anatomical features in an unsupervised manner. Lastly, we both empirically and theoretically, demonstrate the efficacy of our MONA on three benchmark datasets with different labeled settings, achieving new state-of-the-art under different labeled semi-supervised settings. MONA makes minimal assumptions on domain expertise, and hence constitutes a practical and versatile solution in medical image analysis. We provide the PyTorch-like pseudo-code in supplementary.

Single-Photon Emission Computed Tomography (SPECT) is widely applied for the diagnosis of coronary artery diseases. Low-dose (LD) SPECT aims to minimize radiation exposure but leads to increased image noise. Limited-view (LV) SPECT, such as the latest GE MyoSPECT ES system, enables accelerated scanning and reduces hardware expenses but degrades reconstruction accuracy. Additionally, Computed Tomography (CT) is commonly used to derive attenuation maps ( μ -maps) for attenuation correction (AC) of cardiac SPECT, but it will introduce additional radiation exposure and SPECT-CT misalignments. Although various methods have been developed to solely focus on LD denoising, LV reconstruction, or CT-free AC in SPECT, the solution for simultaneously addressing these tasks remains challenging and under-explored. Furthermore, it is essential to explore the potential of fusing cross-domain and cross-modality information across these interrelated tasks to further enhance the accuracy of each task. Thus, we propose a Dual-Domain Coarse-to-Fine Progressive Network (DuDoCFNet), a multi-task learning method for simultaneous LD denoising, LV reconstruction, and CT-free μ -map generation of cardiac SPECT. Paired dual-domain networks in DuDoCFNet are cascaded using a multi-layer fusion mechanism for cross-domain and cross-modality feature fusion. Two-stage progressive learning strategies are applied in both projection and image domains to achieve coarse-to-fine estimations of SPECT projections and CT-derived μ -maps. Our experiments demonstrate DuDoCFNet's superior accuracy in estimating projections, generating μ -maps, and AC reconstructions compared to existing single- or multi-task learning methods, under various iterations and LD levels. The source code of this work is available at https://github.com/XiongchaoChen/DuDoCFNet-MultiTask.

Automated volumetric meshing of patient-specific heart geometry can help expedite various biomechanics studies, such as post-intervention stress estimation. Prior meshing techniques often neglect important modeling characteristics for successful downstream analyses, especially for thin structures like the valve leaflets. In this work, we present DeepCarve (Deep Cardiac Volumetric Mesh): a novel deformation-based deep learning method that automatically generates patient-specific volumetric meshes with high spatial accuracy and element quality. The main novelty in our method is the use of minimally sufficient surface mesh labels for precise spatial accuracy and the simultaneous optimization of isotropic and anisotropic deformation energies for volumetric mesh quality. Mesh generation takes only 0.13 seconds/scan during inference, and each mesh can be directly used for finite element analyses without any manual post-processing. Calcification meshes can also be subsequently incorporated for increased simulation accuracy. Numerous stent deployment simulations validate the viability of our approach for large-batch analyses. Our code is available at https://github.com/danpak94/Deep-Cardiac-Volumetric-Mesh.

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

Unsupervised Domain Adaptation (UDA), which aligns the labeled source distribution to the unlabeled target distribution, has shown remarkable achievement in the medical image segmentation task. Previous UDA methods unilaterally consider the global distribution alignment through explicit category-based loss while good separation and discrimination of class are insufficiently explored, resulting in the sub-aligned distribution across domains. In this paper, we propose cross-prototype contrastive learning method (CPCL) for UDA segmentation through class centroid alignment. Specifically, to reduce the intra-class distance and increase the inter-class distance, we first introduce prototype-feature contrastive learning to align the pixel-level features and the same-class global prototype across domains. Secondly, we further present prototype-prototype contrastive learning to align the same class prototypes between the source domain and target domain for compact category centroid and better global domain distribution alignment. Extensive experiments on two public cardiac datasets demonstrate that the proposed CPCL achieves superior domain adaptation performance as compared with the state-of-the-art.

For medical image segmentation, contrastive learning is the dominant practice to improve the quality of visual representations by contrasting semantically similar and dissimilar pairs of samples. This is enabled by the observation that without accessing ground truth labels, negative examples with truly dissimilar anatomical features, if sampled, can significantly improve the performance. In reality, however, these samples may come from similar anatomical features and the models may struggle to distinguish the minority tail-class samples, making the tail classes more prone to misclassification, both of which typically lead to model collapse. In this paper, we propose $\texttt{ARCO}$, a semi-supervised contrastive learning (CL) framework with stratified group theory for medical image segmentation. In particular, we first propose building $\texttt{ARCO}$ through the concept of variance-reduced estimation, and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks with extremely limited labels. Furthermore, we theoretically prove these sampling techniques are universal in variance reduction. Finally, we experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings, and our methods consistently outperform state-of-the-art semi-supervised methods. Additionally, we augment the CL frameworks with these sampling techniques and demonstrate significant gains over previous methods. We believe our work is an important step towards semi-supervised medical image segmentation by quantifying the limitation of current self-supervision objectives for accomplishing such challenging safety-critical tasks.

Multi-contrast MRI (MC-MRI) captures multiple complementary imaging modalities to aid in radiological decision-making. Given the need for lowering the time cost of multiple acquisitions, current deep accelerated MRI reconstruction networks focus on exploiting the redundancy between multiple contrasts. However, existing works are largely supervised with paired data and/or prohibitively expensive fully-sampled MRI sequences. Further, reconstruction networks typically rely on convolutional architectures which are limited in their capacity to model long-range interactions and may lead to suboptimal recovery of fine anatomical detail. To these ends, we present a dual-domain self-supervised transformer (DSFormer) for accelerated MC-MRI reconstruction. DSFormer develops a deep conditional cascade transformer (DCCT) consisting of cascaded Swin transformer reconstruction networks (SwinRN) trained under two deep conditioning strategies to enable MC-MRI information sharing. We further use a dual-domain (image and k-space) self-supervised learning strategy for DCCT to alleviate the costs of acquiring fully sampled training data. DSFormer generates high-fidelity reconstructions which outperform current fully-supervised baselines and approach the performance of full supervision.

Myocardial perfusion imaging (MPI) using single-photon emission computed tomography (SPECT) is widely applied for the diagnosis of cardiovascular diseases. Attenuation maps (μ-maps) derived from computed tomography (CT) are utilized for attenuation correction (AC) to improve the diagnostic accuracy of cardiac SPECT. However, in clinical practice, SPECT and CT scans are acquired sequentially, potentially inducing misregistration between the two images and further producing AC artifacts. Conventional intensity-based registration methods show poor performance in the cross-modality registration of SPECT and CT-derived μ-maps since the two imaging modalities might present totally different intensity patterns. Deep learning has shown great potential in medical imaging registration. However, existing deep learning strategies for medical image registration encoded the input images by simply concatenating the feature maps of different convolutional layers, which might not fully extract or fuse the input information. In addition, deep-learning-based cross-modality registration of cardiac SPECT and CT-derived μ-maps has not been investigated before. In this paper, we propose a novel Dual-Channel Squeeze-Fusion-Excitation (DuSFE) co-attention module for the cross-modality rigid registration of cardiac SPECT and CT-derived μ-maps. DuSFE is designed based on the co-attention mechanism of two cross-connected input data streams. The channel-wise or spatial features of SPECT and μ-maps are jointly encoded, fused, and recalibrated in the DuSFE module. DuSFE can be flexibly embedded at multiple convolutional layers to enable gradual feature fusion in different spatial dimensions. Our studies using clinical patient MPI studies demonstrated that the DuSFE-embedded neural network generated significantly lower registration errors and more accurate AC SPECT images than existing methods. We also showed that the DuSFE-embedded network did not over-correct or degrade the registration performance of motion-free cases. The source code of this work is available at https://github.com/XiongchaoChen/DuSFE_CrossRegistration.

Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical textures. In this work, we present CASTformer, a novel type of adversarial transformers, for 2D medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CASTformer dramatically outperforms previous state-of-the-art transformer-based approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in Dice over previous models. Further qualitative experiments provide a more detailed picture of the model’s inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CASTformer a strong starting point for downstream medical image analysis tasks.

To reduce the potential risk of radiation to the patient, low-dose computed tomography (LDCT) has been widely adopted in clinical practice for reconstructing cross-sectional images using sinograms with reduced x-ray flux. The LDCT image quality is often degraded by different levels of noise depending on the low-dose protocols. The image quality will be further degraded when the patient has metallic implants, where the image suffers from additional streak artifacts along with further amplified noise levels, thus affecting the medical diagnosis and other CT-related applications. Previous studies mainly focused either on denoising LDCT without considering metallic implants or full-dose CT metal artifact reduction (MAR). Directly applying previous LDCT or MAR approaches to the issue of simultaneous metal artifact reduction and low-dose CT (MARLD) may yield sub-optimal reconstruction results. In this work, we develop a dual-domain under-to-fully-complete progressive restoration network, called DuDoUFNet, for MARLD. Our DuDoUFNet aims to reconstruct images with substantially reduced noise and artifact by progressive sinogram to image domain restoration with a two-stage progressive restoration network design. Our experimental results demonstrate that our method can provide high-quality reconstruction, superior to previous LDCT and MAR methods under various low-dose and metal settings.

Automated segmentation in medical image analysis is a challenging task that requires a large amount of manually labeled data. However, most existing learning-based approaches usually suffer from limited manually annotated medical data, which poses a major practical problem for accurate and robust medical image segmentation. In addition, most existing semi-supervised approaches are usually not robust compared with the supervised counterparts, and also lack explicit modeling of geometric structure and semantic information, both of which limit the segmentation accuracy. In this work, we present SimCVD, a simple contrastive distillation framework that significantly advances state-of-the-art voxel-wise representation learning. We first describe an unsupervised training strategy, which takes two views of an input volume and predicts their signed distance maps of object boundaries in a contrastive objective, with only two independent dropout as mask. This simple approach works surprisingly well, performing on the same level as previous fully supervised methods with much less labeled data. We hypothesize that dropout can be viewed as a minimal form of data augmentation and makes the network robust to representation collapse. Then, we propose to perform structural distillation by distilling pair-wise similarities. We evaluate SimCVD on two popular datasets: the Left Atrial Segmentation Challenge (LA) and the NIH pancreas CT dataset. The results on the LA dataset demonstrate that, in two types of labeled ratios (i.e., 20% and 10%), SimCVD achieves an average Dice score of 90.85% and 89.03% respectively, a 0.91% and 2.22% improvement compared to previous best results. Our method can be trained in an end-to-end fashion, showing the promise of utilizing SimCVD as a general framework for downstream tasks, such as medical image synthesis, enhancement, and registration.

We propose ACProp (Asynchronous-centering-Prop), an adaptive optimizer which combines centering of second momentum and asynchronous update (e.g. for $t$-th update, denominator uses information up to step $t-1$, while numerator uses gradient at $t$-th step). ACProp has both strong theoretical properties and empirical performance. With the example by Reddi et al. (2018), we show that asynchronous optimizers (e.g. AdaShift, ACProp) have weaker convergence condition than synchronous optimizers (e.g. Adam, RMSProp, AdaBelief); within asynchronous optimizers, we show that centering of second momentum further weakens the convergence condition. We demonstrate that ACProp has a convergence rate of $O(\frac{1}{\sqrt{T}})$ for the stochastic non-convex case, which matches the oracle rate and outperforms the $O(\frac{logT}{\sqrt{T}})$ rate of RMSProp and Adam. We validate ACProp in extensive empirical studies: ACProp outperforms both SGD and other adaptive optimizers in image classification with CNN, and outperforms well-tuned adaptive optimizers in the training of various GAN models, reinforcement learning and transformers. To sum up, ACProp has good theoretical properties including weak convergence condition and optimal convergence rate, and strong empirical performance including good generalization like SGD and training stability like Adam. We provide the implementation at \url{ https://github.com/juntang-zhuang/ACProp-Optimizer}.

Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in deep learning are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, etc. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.

Reliable motion estimation and strain analysis using 3D+ time echocardiography (4DE) for localization and characterization of myocardial injury is valuable for early detection and targeted interventions. However, motion estimation is difficult due to the low-SNR that stems from the inherent image properties of 4DE, and intelligent regularization is critical for producing reliable motion estimates. In this work, we incorporated the notion of domain adaptation into a supervised neural network regularization framework. We first propose a semi-supervised Multi-Layered Perceptron (MLP) network with biomechanical constraints for learning a latent representation that is shown to have more physiologically plausible displacements. We extended this framework to include a supervised loss term on synthetic data and showed the effects of biomechanical constraints on the network's ability for domain adaptation. We validated the semi-supervised regularization method on in vivo data with implanted sonomicrometers. Finally, we showed the ability of our semi-supervised learning regularization approach to identify infarct regions using estimated regional strain maps with good agreement to manually traced infarct regions from postmortem excised hearts.

In positron emission tomography (PET), gating is commonly utilized to reduce respiratory motion blurring and to facilitate motion correction methods. In application where low-dose gated PET is useful, reducing injection dose causes increased noise levels in gated images that could corrupt motion estimation and subsequent corrections, leading to inferior image quality. To address these issues, we propose MDPET, a unified motion correction and denoising adversarial network for generating motion-compensated low-noise images from low-dose gated PET data. Specifically, we proposed a Temporal Siamese Pyramid Network (TSP-Net) with basic units made up of 1.) Siamese Pyramid Network (SP-Net), and 2.) a recurrent layer for motion estimation among the gates. The denoising network is unified with our motion estimation network to simultaneously correct the motion and predict a motion-compensated denoised PET reconstruction. The experimental results on human data demonstrated that our MDPET can generate accurate motion estimation directly from low-dose gated images and produce high-quality motion-compensated low-noise reconstructions. Comparative studies with previous methods also show that our MDPET is able to generate superior motion estimation and denoising performance. Our code is available at https://github.com/bbbbbbzhou/MDPET.

Limited view tomographic reconstruction aims to reconstruct a tomographic image from a limited number of projection views arising from sparse view or limited angle acquisitions that reduce radiation dose or shorten scanning time. However, such a reconstruction suffers from severe artifacts due to the incompleteness of sinogram. To derive quality reconstruction, previous methods use UNet-like neural architectures to directly predict the full view reconstruction from limited view data; but these methods leave the deep network architecture issue largely intact and cannot guarantee the consistency between the sinogram of the reconstructed image and the acquired sinogram, leading to a non-ideal reconstruction. In this work, we propose a cascaded residual dense spatial-channel attention network consisting of residual dense spatial-channel attention networks and projection data fidelity layers. We evaluate our methods on two datasets. Our experimental results on AAPM Low Dose CT Grand Challenge datasets demonstrate that our algorithm achieves a consistent and substantial improvement over the existing neural network methods on both limited angle reconstruction and sparse view reconstruction. In addition, our experimental results on Deep Lesion datasets demonstrate that our method is able to generate high-quality reconstruction for 8 major lesion types.

The empirical performance of neural ordinary differential equations (NODEs) is significantly inferior to discrete-layer models on benchmark tasks (e.g. image classification). We demonstrate an explanation is the inaccuracy of existing gradient estimation methods: the adjoint method has numerical errors in reverse-mode integration; the naive method suffers from a redundantly deep computation graph. We propose the Adaptive Checkpoint Adjoint (ACA) method: ACA applies a trajectory checkpoint strategy which records the forward- mode trajectory as the reverse-mode trajectory to guarantee accuracy; ACA deletes redundant components for shallow computation graphs; and ACA supports adaptive solvers. On image classification tasks, compared with the adjoint and naive method, ACA achieves half the error rate in half the training time; NODE trained with ACA outperforms ResNet in both accuracy and test-retest reliability. On time-series modeling, ACA outperforms competing methods. Furthermore, NODE with ACA can incorporate physical knowledge to achieve better accuracy.

Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g.~Adam) and accelerated schemes (e.g.~stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability. We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer.

Deep learning models have shown their advantage in many different tasks, including neuroimage analysis. However, to effectively train a high-quality deep learning model, the aggregation of a significant amount of patient information is required. The time and cost for acquisition and annotation in assembling, for example, large fMRI datasets make it difficult to acquire large numbers at a single site. However, due to the need to protect the privacy of patient data, it is hard to assemble a central database from multiple institutions. Federated learning allows for population-level models to be trained without centralizing entities' data by transmitting the global model to local entities, training the model locally, and then averaging the gradients or weights in the global model. However, some studies suggest that private information can be recovered from the model gradients or weights. In this work, we address the problem of multi-site fMRI classification with a privacy-preserving strategy. To solve the problem, we propose a federated learning approach, where a decentralized iterative optimization algorithm is implemented and shared local model weights are altered by a randomization mechanism. Considering the systemic differences of fMRI distributions from different sites, we further propose two domain adaptation methods in this federated learning formulation. We investigate various practical aspects of federated model optimization and compare federated learning with alternative training strategies. Overall, our results demonstrate that it is promising to utilize multi-site data without data sharing to boost neuroimage analysis performance and find reliable disease-related biomarkers. Our proposed pipeline can be generalized to other privacy-sensitive medical data analysis problems. Our code is publicly available at: https://github.com/xxlya/Fed_ABIDE/.

Imaging of the human body using a number of different modalities has revolutionized the field of medicine over the past several decades and continues to grow at a rapid pace[2]. More than ever, previously unknown information about biology and disease is being unveiled at a range of spatiotemporal scales. Although results and clinical adoption of strategies related to the computational and quantitative analysis of the images have lagged behind development of image acquisition approaches, there has been a noticeable increase of effort and interest in these areas in recent years[6]. This special issue aims to define and highlight some of the “hot” newer ideas that are in biomedical imaging and analysis, intending to shine a light on where the field might move in the next several decades, and focuses on emphasizing where electrical engineers have been involved and could potentially have the most impact. These areas include image acquisition physics, image/signal processing, and image analysis, including pattern recognition and machine learning. This issue focuses on two themes common in much of this effort: first, engineers and computer scientists have found that the information contained in medical images, when viewed through image-based vector spaces, is generally quite sparse. This observation has been transformative in many ways and is quite pervasive in the articles we include here. Second, medical imaging is one of the largest producers of “big data,” and, data-driven machinelearning techniques (e.g., deep learning) are gaining significant attention because improved performance over previous approaches. Thus, data-driven techniques, e.g., formation via image reconstruction[11]and image analysis via deep learning[8],[9], are gaining momentum in their development.

In this project, our goal is to develop a method for interpreting how a neural network makes layer-by-layer embedded decisions when trained for a classification task, and also to use this insight for improving the model performance. To do this, we first approximate the distribution of the image representations in these embeddings using random forest models, the output of which, termed embedding outputs, are used for measuring how the network classifies each sample. Next, we design a pipeline to use this layer embedding output to calibrate the original model output for improved probability calibration and classification. We apply this two-steps method in a fully convolutional neural network trained for a liver tissue classification task on our institutional dataset that contains 20 3D multi-parameter MR images for patients with hepatocellular carcinoma, as well as on a public dataset with 131 3D CT images. The results show that our method is not only able to provide visualizations that are easy to interpret, but that the embedded decision-based information is also useful for improving model performance in terms of probability calibration and classification, achieving the best performance compared to other baseline methods. Moreover, this method is computationally efficient, easy to implement, and robust to hyper-parameters.

We propose a unified Bayesian framework to detect both hyper- and hypo-active communities within whole-brain fMRI data. Specifically, our model identifies dense subgraphs that exhibit population-level differences in functional synchrony between a control and clinical group. We derive a variational EM algorithm to solve for the latent posterior distributions and parameter estimates, which subsequently inform us about the afflicted network topology. We demonstrate that our method provides valuable insights into the neural mechanisms underlying social dysfunction in autism, as verified by the Neurosynth meta-analytic database. In contrast, both univariate testing and community detection via recursive edge elimination fail to identify stable functional communities associated with the disorder.

In the above-named document [ibid, vol. 33, no. 6, pp. 1275–1289, Jun. 2014], the funding information should have appeared as follows: "This work was supported in part by the National Institutes of Health (The National Heart, Lung, and Blood Institute) through these awards: R01HL082640, 5T32HL098069, and R01HL121226. The work of B. A. Lin was supported by an ASE Foundation Career Development Award."

In order to quantitatively analyze biological images and study underlying mechanisms of the cellular and subcellular processes, it is often required to track a large number of particles involved in these processes. Manual tracking can be performed by the biologists, but the workload is very heavy. In this paper, we present an automatic particle tracking method for analyzing an essential subcellular process, namely clathrin mediated endocytosis. The framework of the tracking method is an extension of the classical multiple hypothesis tracking (MHT), and it is designed to manage trajectories, solve data association problems, and handle pseudo-splitting/merging events. In the extended MHT framework, particle tracking becomes evaluating two types of hypotheses. The first one is the trajectory-related hypothesis, to test whether a recovered trajectory is correct, and the second one is the observation-related hypothesis, to test whether an observation from an image belongs to a real particle. Here, an observation refers to a detected particle and its feature vector. To detect the particles in 2D fluorescence images taken using total internal reflection microscopy, the images are segmented into regions, and the features of the particles are obtained by fitting Gaussian mixture models into each of the image regions. Specific models are developed according to the properties of the particles. The proposed tracking method is demonstrated on synthetic data under different scenarios and applied to real data.

Quantitative analysis of left ventricular deformation can provide valuable information about the extent of disease as well as the efficacy of treatment. In this work, we develop an adaptive multi-level compactly supported radial basis approach for deformation analysis in 3D+time echocardiography. Our method combines displacement information from shape tracking of myocardial boundaries (derived from B-mode data) with mid-wall displacements from radio-frequency-based ultrasound speckle tracking. We evaluate our methods on open-chest canines (N=8) and show that our combined approach is better correlated to magnetic resonance tagging-derived strains than either individual method. We also are able to identify regions of myocardial infarction (confirmed by postmortem analysis) using radial strain values obtained with our approach.

Active contours are widely used in image segmentation. To cope with missing or misleading features in images, researchers have introduced various ways to model the prior of shapes and use the prior to constrain active contours. However, the shape prior is usually learnt from a large set of annotated data, which is not always accessible in practice. Moreover, it is often doubted that the existing shapes in the training set will be sufficient to model the new instance in the testing image. In this paper, we propose to use the group similarity of object shapes in multiple images as a prior to aid segmentation, which can be interpreted as an unsupervised approach of shape prior modeling. We show that the rank of the matrix consisting of multiple shapes is a good measure of the group similarity of the shapes, and the nuclear norm minimization is a simple and effective way to impose the proposed constraint on existing active contour models. Moreover, we develop a fast algorithm to solve the proposed model by using the accelerated proximal method. Experiments using echocardiographic image sequences acquired from acute canine experiments demonstrate that the proposed method can consistently improve the performance of active contour models and increase the robustness against image defects such as missing boundaries.

During neurosurgery, nonrigid brain deformation may affect the reliability of tissue localization based on preoperative images. To provide accurate surgical guidance in these cases, preoperative images must be updated to reflect the intraoperative brain. This can be accomplished by warping these preoperative images using a biomechanical model. Due to the possible complexity of this deformation, intraoperative information is often required to guide the model solution. In this paper, a linear elastic model of the brain is developed to infer volumetric brain deformation associated with measured intraoperative cortical surface displacement. The developed model relies on known material properties of brain tissue, and does not require further knowledge about intraoperative conditions. To provide an initial estimation of volumetric model accuracy, as well as determine the model's sensitivity to the specified material parameters and surface displacements, a realistic brain phantom was developed. Phantom results indicate that the linear elastic model significantly reduced localization error due to brain shift, from > 16 mm to under 5 mm, on average. In addition, though in vivo quantitative validation is necessary, preliminary application of this approach to images acquired during neocortical epilepsy cases confirms the feasibility of applying the developed model to in vivo data.

External beam radiation therapy (EBRT) for the treatment of cancer enables accurate placement of radiation dose on the cancerous region. However, the deformation of soft tissue during the course of treatment, such as in cervical cancer, presents significant challenges for the delineation of the target volume and other structures of interest. Furthermore, the presence and regression of pathologies such as tumors may violate registration constraints and cause registration errors. In this paper, automatic segmentation, nonrigid registration and tumor detection in cervical magnetic resonance (MR) data are addressed simultaneously using a unified Bayesian framework. The proposed novel method can generate a tumor probability map while progressively identifying the boundary of an organ of interest based on the achieved nonrigid transformation. The method is able to handle the challenges of significant tumor regression and its effect on surrounding tissues. The new method was compared to various currently existing algorithms on a set of 36 MR data from six patients, each patient has six T2-weighted MR cervical images. The results show that the proposed approach achieves an accuracy comparable to manual segmentation and it significantly outperforms the existing registration algorithms. In addition, the tumor detection result generated by the proposed method has a high agreement with manual delineation by a qualified clinician.

This paper proposed a new strategy to assess cumulative actual dose by nonrigidly mapping treatment day dose distributions to the planning day space. The mappings were achieved using a novel integrated segmentation and nonrigid registration algorithm used in external beam prostate radiotherapy. By combining segmentation and registration, we can recover the treatment fraction image regions that correspond to the organs of interest by incorporating transformed planning day organs to guide and constrain the segmentation process; and conversely, accurate knowledge of important soft tissue structures will enable us achieve more precise nonrigid registration. The novel algorithm allows the clinician to set tighter planning margins around the target volume in the treatment plan. Clinical application and evaluation of dose delivery show the superiority of proposed method to the procedure currently used in clinical practice, i.e. manual segmentation followed by rigid registration.

Total internal reflection fluorescence (TIRF) microscopy excites a thin evanescent field which theoretically decays exponentially. Each TIRF image is actually the projection of a 3-D volume and hence cannot alone produce an accurate localization of structures in the z-dimension, however, it provides greatly improved axial resolution for biological samples. Multiple angle-TIRF microscopy allows controlled variation of the incident angle of the illuminating laser beam, thus generating a set of images of different penetration depths with the potential to reconstruct the 3-D volume of the sample. With the ultimate goal to quantify important biological parameters of microtubules, we present a method to reconstruct 3-D position and orientation of microtubules based on multi-angle TIRF data, as well as experimental calibration of the actual decay function of the evanescent field at each angle. We validate our method using computer simulations, by creating a phantom simulating the curvilinear characteristics of microtubules and project the artificially constructed volume into a set of TIRF image for different penetration depth. The reconstructed depth information for the phantom data is shown to be accurate and robust to noise. We apply our method to microtubule TIRF images of PtK(2) cells in vivo. By comparing microtubule curvatures of the reconstruction results and several electron microscopy (EM) images of vertically sliced sample of microtubules, we find that the curvature statistics of our reconstruction agree well with the ground truth (EM data). Quantifying the distribution of microtubule curvature reveals an interesting discovery that microtubules can buckle and form local bendings of considerably small radius of curvature which is also visually spotted on the EM images, while microtubule bendings on a larger scale generally have a much larger radius and cannot bear the stress of a large curvature. The presented method has the potential to provide a reliable tool for 3-D reconstruction and tracking of microtubules.

During neurosurgery, nonrigid brain deformation prevents preoperatively-acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track intraoperative cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration from the surface deformation estimation, a framework that can solve for disparate and often competing variables is needed. Game theory, which was developed to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, game theory is applied to cortical surface tracking during neocortical epilepsy surgery and used to infer information about the physical processes of brain surface deformation and image acquisition. The method is successfully applied to eight in vivo cases, resulting in an 81% decrease in mean surface displacement error. This includes a case in which some of the initial camera calibration parameters had errors of 70%. Additionally, the advantages of using a game theoretic approach in neocortical epilepsy surgery are clearly demonstrated in its robustness to initial conditions.

Statistical models have shown considerable promise as a basis for segmenting and interpreting cardiac images. While a variety of statistical models have been proposed to improve the segmentation results, most of them are either static models (SMs), which neglect the temporal dynamics of a cardiac sequence, or generic dynamical models (GDMs), which are homogeneous in time and neglect the intersubject variability in cardiac shape and deformation. In this paper, we develop a subject-specific dynamical model (SSDM) that simultaneously handles temporal dynamics (intrasubject variability) and intersubject variability. We also propose a dynamic prediction algorithm that can progressively identify the specific motion patterns of a new cardiac sequence based on the shapes observed in past frames. The incorporation of this SSDM into the segmentation framework is formulated in a recursive Bayesian framework. It starts with a manual segmentation of the first frame, and then segments each frame according to intensity information from the current frame as well as the prediction from past frames. In addition, to reduce error propagation in sequential segmentation, we take into account the periodic nature of cardiac motion and perform segmentation in both forward and backward directions. We perform "leave-one-out" test on 32 canine sequences and 22 human sequences, and compare the experimental results with those from SM, GDM, and active appearance motion model (AAMM). Quantitative analysis of the experimental results shows that SSDM outperforms SM, GDM, and AAMM by having better global and local consistencies with manual segmentation. Moreover, we compare the segmentation results from forward and forward-backward segmentation. Quantitative evaluation shows that forward-backward segmentation suppresses the propagation of segmentation errors.

The development of methods to accurately and reproducibly recover useful quantitative information from medical images is often hampered by uncertainties in handling the data related to: image acquisition parameters, the variability of normal human anatomy and physiology, the presence of disease or other abnormal conditions, and a variety of other factors. This talk will review image analysis strategies that make use of models based on geometrical and physical/biomechanical information to help constrain the range of possible solutions in the presence of such uncertainty. The discussion will be focused by looking primarily at several problem areas in the realms of neuroanatomical structure analysis and cardiac function analysis, along with some work in cellular image analysis, with an emphasis on image segmentation and motion/deformation tracking. The presentation will include a description of the problem areas and visual examples of the image datasets being used, an overview of the mathematical techniques involved and a presentation of results obtained when analyzing actual patient image data using these methods. Emphasis will be placed on how image-derived information and appropriate modeling can be used together to address the image analysis and processing problems noted above.

Statistical model-based segmentation of the left ventricle from cardiac images has received considerable attention in recent years. While a variety of statistical models have been shown to improve segmentation results, most of them are either static models (SM) which neglect the temporal coherence of a cardiac sequence or generic dynamical models (GDM) which neglect the inter-subject variability of cardiac shapes and deformations. In this paper, we use a subject-specific dynamical model (SSDM) that handles inter-subject variability and temporal dynamics (intra-subject variability) simultaneously. It can progressively identify the specific motion patterns of a new cardiac sequence based on the segmentations observed in the past frames. We formulate the integration of the SSDM into the segmentation process in a recursive Bayesian framework in order to segment each frame based on the intensity information from the current frame and the prediction from the past frames. We perform "Leave-one-out" test on 32 sequences to validate our approach. Quantitative analysis of experimental results shows that the segmentation with the SSDM outperforms those with the SM and GDM by having better global and local consistencies with the manual segmentation.

This paper presents a new shape registration algorithm that establishes "meaningful correspondence " between objects, in that it preserves the local shape correspondence between the source and target objects. By observing that an object's skeleton corresponds to its local shape peaks, we use skeleton to characterize the local shape of the source and target objects. Unlike traditional graph-based skeleton matching algorithms that focus on matching skeletons alone and ignore the overall alignment of the boundaries, our algorithm is formulated in a variational framework which aligns local shape by registering two potential fields that are associated with skeletons. Also, we add a boundary constraint term to the energy functional, such that our algorithm can be applied to match bulky objects where skeleton and boundary are far away to each other. To increase the robustness of our algorithm, we incorporate M-estimator and dynamic pruning algorithm to form a feedback system that eliminates local shape outliers caused by nonrigid deformation, occlusion, and missing parts. Experiments on 2D binary shapes and 3D cardiac sequences validate the accuracy and robustness of this algorithm.

During neurosurgery, nonrigid brain deformation prevents preoperatively acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration and surface deformation, a framework is needed which can solve for disparate and often competing variables. Game theory, which was developed specifically to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, we apply game theory to cortical surface tracking and use it to infer information about the physical processes of brain deformation and image acquisition.

A novel method for the segmentation of multiple objects from three-dimensional (3-D) medical images using interobject constraints is presented. Our method is motivated by the observation that neighboring structures have consistent locations and shapes that provide configurations and context that aid in segmentation. We define a maximum a posteriori (MAP) estimation framework using the constraining information provided by neighboring objects to segment several objects simultaneously. We introduce a representation for the joint density function of the neighbor objects, and define joint probability distributions over the variations of the neighboring shape and position relationships of a set of training images. In order to estimate the MAP shapes of the objects, we formulate the model in terms of level set functions, and compute the associated Euler-Lagrange equations. The contours evolve both according to the neighbor prior information and the image gray level information. This method is useful in situations where there is limited interobject information as opposed to robust global atlases. In addition, we compare our level set representation of the object shape to the point distribution model. Results and validation from experiments on synthetic data and medical imagery in two-dimensional and 3-D are demonstrated.

For patient setup verification in external beam radiotherapy (EBRT) of prostate cancer, we developed an information theoretic registration framework, called the minimax entropy registration framework, to simultaneously and iteratively segment portal images and register them to three-dimensional (3-D) computed tomography (CT) image data. The registration framework has two steps, the max step and the min step, and evaluates appropriate entropies to estimate segmentations of the portal images and to find the transformation parameters. In the initial version of the algorithm (Bansal et al. 1999), we assumed image pixels to be independently distributed, an assumption not true in general. Thus, to better segment the portal images and to improve the accuracy of the estimated registration parameters, in this initial formulation of the problem, the correlation among pixel intensities is modeled using a one-dimensional Markov random process. Line processes are incorporated into the model to improve the estimation of segmentation of the portal images. In the max step, the principle of maximum entropy is invoked to estimate the probability distribution on the segmentations. The estimated distribution is then incorporated into the min step to estimate the registration parameters. Performance of the proposed framework is evaluated and compared to that of a mutual information-based registration algorithm using both simulated and real patient data. In the proposed registration framework, registration of the 3-D CT image and the portal images is guided by an estimated segmentation of the pelvic bone. However, as the prostate can move with respect to the pelvic structure, further localization of the prostate using ultrasound image data is required, an issue to be further explored in future.

Medical image analysis has grown and evolved tremendously in the last 30 years. The distinctive nature of the problems encountered have led to the development of a significant body of work addressing such issues as fully three-dimensional data,

The quantitative estimation of regional cardiac deformation from three-dimensional (3-D) image sequences has important clinical implications for the assessment of viability in the heart wall. We present here a generic methodology for estimating soft tissue deformation which integrates image-derived information with biomechanical models, and apply it to the problem of cardiac deformation estimation. The method is image modality independent. The images are segmented interactively and then initial correspondence is established using a shape-tracking approach. A dense motion field is then estimated using a transversely isotropic, linear-elastic model, which accounts for the muscle fiber directions in the left ventricle. The dense motion field is in turn used to calculate the deformation of the heart wall in terms of strain in cardiac specific directions. The strains obtained using this approach in open-chest dogs before and after coronary occlusion, exhibit a high correlation with strains produced in the same animals using implanted markers. Further, they show good agreement with previously published results in the literature. This proposed method provides quantitative regional 3-D estimates of heart deformation.

We present a new theoretical result for the problem of surface reconstruction from stereo images. For a given initial seed point, i.e. for a pair of corresponding points in the left and right image, the proposed algorithm grows the surface without directly computing the point correspondences. The method assumes the Lambertian surface reflectance model. Our approach is based on a surface normal calculation from the left and right image gradients. Knowing the surface normal, the algorithm grows the surface in the directions defined by the tangent plane. The algorithm is independent of the camera model, and requires placement of an initial seed point for each surface to be reconstructed. Technical problems associated with errors in the image gradient estimates and camera calibration are discussed and a solution is suggested. In addition to this algorithm, we present a theoretical result that permits one to track surfaces deforming in time, which is often encountered in medical applications (e.g. brain surface deforms during the surgery). These methods of surface reconstruction and deformable surface tracking are applied to the particular problem of brain shift, commonly recognized as one of the main source of errors in surgical navigation systems used in neurosurgery. We also suggest a way to overcome problems associated with brain surface specularities caused by fluids on the brain surface and lights in the operating room. We conclude with experimental results on real brain images and show that the surface reconstruction algorithm is robust to the position of the initial seed point.

OBJECTIVE: Currently, when cytopathology images are archived, they are typically stored with a limited text-based description of their content. Such a description inherently fails to quantify the properties of an image and refers to an extremely small fraction of its information content. This paper describes a method for automatically indexing images of individual cells and their associated diagnoses by computationally derived cell descriptors. This methodology may serve to better index data contained in digital image databases, thereby enabling cytologists and pathologists to cross-reference cells of unknown etiology or nature. DESIGN: The indexing method, implemented in a program called PathMaster, uses a series of computer-based feature extraction routines. Descriptors of individual cell characteristics generated by these routines are employed as indexes of cell morphology, texture, color, and spatial orientation. MEASUREMENTS: The indexing fidelity of the program was tested after populating its database with images of 152 lymphocytes/lymphoma cells captured from lymph node touch preparations stained with hematoxylin and eosin. Images of "unknown" lymphoid cells, previously unprocessed, were then submitted for feature extraction and diagnostic cross-referencing analysis. RESULTS: PathMaster listed the correct diagnosis as its first differential in 94 percent of recognition trials. In the remaining 6 percent of trials, PathMaster listed the correct diagnosis within the first three "differentials." CONCLUSION: PathMaster is a pilot cell image indexing program/search engine that creates an indexed reference of images. Use of such a reference may provide assistance in the diagnostic/prognostic process by furnishing a prioritized list of possible identifications for a cell of uncertain etiology.

The analysis of medical images has been woven into the fabric of the pattern analysis and machine intelligence (PAMI) community since the earliest days of these Transactions. Initially, the efforts in this area were seen as applying pattern analysis and computer vision techniques to another interesting dataset. However, over the last two to three decades, the unique nature of the problems presented within this area of study have led to the development of a new discipline in its own right. Examples of these include: the types of image information that are acquired, the fully three-dimensional image data, the nonrigid nature of object motion and deformation, and the statistical variation of both the underlying normal and abnormal ground truth. In this paper, we look at progress in the field over the last 20 years and suggest some of the challenges that remain for the years to come.

A robust, flexible system for tracking the point to point nonrigid motion of the left ventricular (LV) endocardial wall in image sequences has been developed. This system is unique in its ability to model motion trajectories across multiple frames. The foundation of this system is an adaptive transversal filter based on the recursive least-squares algorithm. This filter facilitates the integration of models for periodicity and proximal smoothness as appropriate using a contour-based description of the object's boundaries. A set of correspondences between contours and an associated set of correspondence quality measures comprise the input to the system. Frame-to-frame relationships from two different frames of reference are derived and analyzed using synthetic and actual images. Two multiframe temporal models, both based on a sum of sinusoids, are derived. Illustrative examples of the system's output are presented for quantitative analysis. Validation of the system is performed by comparing computed trajectory estimates with the trajectories of physical markers implanted in the LV wall. Sample case studies of marker trajectory comparisons are presented. Ensemble statistics from comparisons with 15 marker trajectories are acquired and analyzed. A multiframe temporal model without spatial periodicity constraints was determined to provide excellent performance with the least computational cost. A multiframe spatiotemporal model provided the best performance based on statistical standard deviation, although at significant computational expense.

We propose and validate the hypothesis that we can use differential shape properties of the myocardial surfaces to recover dense field motion from standard three-dimensional (3-D) image sequences (MRI and CT). Quantitative measures of left ventricular regional function can be further inferred from the point correspondence maps. The noninvasive, algorithm-derived results are validated on two levels. First, the motion trajectories are compared to those of implanted imaging-opaque markers of a canine model in two imaging modalities, where subpixel accuracy is achieved. Second, the validity of using motion parameters (path length and thickness changes) for detecting myocardial injury area is tested by comparing algorithms derived results to postmortem analysis TTC staining of myocardial tissue, where the achieved Pearson product-moment correlation value is 0.968.

Mapping of functional magnetic resonance imaging (fMRI) to conventional anatomical MRI is a valuable step in the interpretation of fMRI activations. One of the main limits on the accuracy of this alignment arises from differences in the geometric distortion induced by magnetic field inhomogeneity. This paper describes an approach to the registration of echo planar image (EPI) data to conventional anatomical images which takes into account this difference in geometric distortion. We make use of an additional spin echo EPI image and use the known signal conservation in spin echo distortion to derive a specialized multimodality nonrigid registration algorithm. We also examine a plausible modification using log-intensity evaluation of the criterion to provide increased sensitivity in areas of low EPI signal. A phantom-based imaging experiment is used to evaluate the behavior of the different criteria, comparing nonrigid displacement estimates to those provided by a imagnetic field mapping acquisition. The algorithm is then applied to a range of nine brain imaging studies illustrating global and local improvement in the anatomical alignment and localization of fMRI activations.

We have developed a new mutual information-based registration method for matching unlabeled point features. In contrast to earlier mutual information-based registration methods, which estimate the mutual information using image intensity information, our approach uses the point feature location information. A novel aspect of our approach is the emergence of correspondence (between the two sets of features) as a natural by-product of joint density estimation. We have applied this algorithm to the problem of geometric alignment of primate autoradiographs. We also present preliminary results on three-dimensional robust matching of sulci derived from anatomical magnetic resonance images. Finally, we present an experimental comparison between the mutual information approach and other recent approaches which explicitly parameterize feature correspondence.

Robust segmentation of structures from an image is essential for a variety of image analysis problems. However, the conventional methods of region-based segmentation and gradient-based boundary finding are often frustrated by poor image quality. Here we propose a method to integrate the two approaches using game theory in an effort to form a unified approach that is robust to noise and poor initialization. This combines the perceptual notions of complete boundary information using edge data and shape priors with gray-level homogeneity using two computational modules. The novelty of the method is that this is a bidirectional framework, whereby both computational modules improve their results through mutual information sharing. A number of experiments were performed both on synthetic datasets and datasets of real images to evaluate the new approach and it is shown that the integrated method typically performs better than conventional gradient-based boundary finding.

The cortex is the outermost thin layer of gray matter in the brain; geometric measurement of the cortex helps in understanding brain anatomy and function. In the quantitative analysis of the cortex from MR images, extracting the structure and obtaining a representation for various measurements are key steps. While manual segmentation is tedious and labor intensive, automatic reliable efficient segmentation and measurement of the cortex remain challenging problems, due to its convoluted nature. Here we present a new approach of coupled-surfaces propagation, using level set methods to address such problems. Our method is motivated by the nearly constant thickness of the cortical mantle and takes this tight coupling as an important constraint. By evolving two embedded surfaces simultaneously, each driven by its own image-derived information while maintaining the coupling, a final representation of the cortical bounding surfaces and an automatic segmentation of the cortex are achieved. Characteristics of the cortex, such as cortical surface area, surface curvature, and cortical thickness, are then evaluated. The level set implementation of surface propagation offers the advantage of easy initialization, computational efficiency, and the ability to capture deep sulcal folds. Results and validation from various experiments on both simulated and real three-dimensional (3-D) MR images are provided.

The problem of segmenting a volumetric layer of finite thickness is encountered in several important areas within medical image analysis. Key examples include the extraction of the cortical gray matter of the brain and the left ventricle myocardium of the heart. The coupling between the two bounding surfaces of such a layer provides important information that helps to solve the segmentation problem. Here we propose a new approach of coupled surfaces propagation via level set methods, which takes into account coupling as an important constraint. By evolving two embedded surfaces simultaneously, each driven by its own image-derived information while maintaining the coupling, we capture a representation of the two bounding surfaces and achieve automatic segmentation on the layer. Characteristic gray level values, instead of image gradient information alone, are incorporated in deriving the useful image information to drive the surface propagation, which enables our approach to capture the homogeneity inside the layer. The level set implementation offers the advantage of easy initialization, computational efficiency and the ability to capture deep folds of the sulci. As a test example, we apply our approach to unedited 3D Magnetic Resonance (MR) brain images. Our algorithm automatically isolates the brain from non-brain structures and recovers the cortical gray matter.

In optic flow based velocity estimation the image brightness constraint equation is used. However, for measurements performed at a certain scale, the brightness constraint equation does not apply. We therefore use a recently developed approach which reconciles optic flow and scale space theory. It specifically incorporates the scale (aperture) of image measurements, leading to a scheme which is essentially different from existing approaches. To obtain a unique velocity field, the data-derived information has to be augmented with physical knowledge. By keeping a strict separation between data-derived and external information, we can locally adapt or modify the user-supplied information without affecting the image-derived information. The two free scale parameters in time and space can be used for attentive vision (selecting particular velocities or objects) and to improve the reliability of velocity estimates.

The wide availability of high resolution magnetic resonance images (MRI) of the brain has facilitated tremendous progress in neuroscience. Accurate automated segmentation and quantification of neuroanatomical structure from such images is crucial for the advancement of the understanding of brain morphology, both in normal variation and in disease. Gradient-based deformable surface finding is a powerful technique for locating structure in three-dimensional images. However, it often suffers from poorly defined edges and noise. This paper proposes a gradient-based deformable surface finding approach that integrates region information. This makes the resulting procedure more robust to noise and improper initialization. In addition, prior shape information may be incorporated. The algorithm uses Gauss's Divergence theorem to find the surface of a homogeneous region-classified area in the image and integrates this with a gray-level gradient-based surface finder. Experimental results on synthetic and MR brain images show that a significant improvement is achieved as a consequence of the use of this extra information. Further, these improvements are achieved with little increase in computational overhead, an advantage derived from the application of Gauss's Divergence theorem.

Information contained in medical images differs considerably from that residing in alphanumeric format. The difference can be attributed to four characteristics: (1) the semantics of medical knowledge extractable from images is imprecise; (2) image information contains form and spatial data, which are not expressible in conventional language; (3) a large part of image information is geometric; (4) diagnostic inferences derived from images rest on an incomplete, continuously evolving model of normality. This paper explores the differentiating characteristics of text versus images and their impact on design of a medical image database intended to allow content-based indexing and retrieval. One strategy for implementing medical image databases is presented, which employs object-oriented iconic queries, semantics by association with prototypes, and a generic schema.

We present a novel method for the geometric alignment of autoradiographs of the brain. The method is based on finding the spatial mapping and the one-to-one correspondences (or homologies) between point features extracted from the images and rejecting non-homologies as outliers. In this way, we attempt to account for the local, natural and artifactual differences between the autoradiograph slices. We have used the resulting automated algorithm on a set of left prefrontal cortex autoradiograph slices, specifically demonstrated its ability to perform point outlier rejection, validated its robustness property using synthetically generated spatial mappings and provided an anecdotal visual comparison with the well-known iterated closest-point (ICP) algorithm. Visualization of a stack of aligned left prefrontal cortex autoradiograph slices is also provided.

An approach for tracking and quantifying the nonrigid, nonuniform motion of the left ventricular (LV) endocardial wall from two-dimensional (2-D) cardiac image sequences, on a point-by-point basis over the entire cardiac cycle, is presented. Given a set of boundaries, motion computation involves first matching local segments on one contour to segments on the next contour in the sequence using a shape-based strategy. Results from the match process are incorporated with a smoothness term into an optimization functional. The global minimum of this functional is found, resulting in a smooth flow field that is consistent with the match data. The computation is performed for all pairs of frames in the temporal sequence and equally sampled points on one contour are tracked throughout the sequence, resulting in a composite flow field over the entire sequence. Two perspectives on characterizing the optimization functional are presented which result in a tradeoff resolved by the confidence in the initial boundary segmentation. Experimental results for contours derived from diagnostic image sequences of three different imaging modalities are presented. A comparison of trajectory estimates with trajectories of gold-standard markers implanted in the LV wall are presented for validation. The results of this comparison confirm that although cardiac motion is a three-dimensional (3-D) problem, two-dimensional (2-D) analysis provides a rich testing ground for algorithm development.

We have addressed the problem of tracking the non-rigid motion of the heart using a sequence of velocity fields and a sequence of contours. The information from both the contours and the dense velocity fields is integrated into a deforming mesh that is placed over the myocardium at one time frame and then tracked over the entire cardiac cycle. The deformation is guided by a smoothing filter that provides a compromise between (i) believing the dense field velocity and the contour data when it is crisp and coherent in a local spatial and temporal sense and (ii) employing a temporally smooth cyclic model of cardiac motion when contour and velocity data are not trustworthy. The method has been carefully evaluated with simulated data and phantom data. Experiments with in vivo data have also been conducted.

Segmentation of (noisy) images containing a complex ensemble of objects is difficult to achieve on the basis of local image information only. It is advantageous to attack the problem of object boundary extraction by a model-based segmentation procedure. Segmentation is achieved by tuning the parameters of the geometrical model in such a way that the boundary template locates and describes the object in the image in an optimal way. The optimality of the solution is based on an objective function taking into account image information as well as the shape of the template. Objective functions in literature are mainly based on the gradient magnitude and a measure describing the smoothness of the template. In this contribution, we propose a new image objective function based on directional gradient information derived from Gaussian smoothed derivatives of the image data. The proposed method is designed to accurately locate an object boundary even in the case of a conflicting object positioned close to the object of interest. We further introduce a new smoothness objective to ensure the physical feasibility of the contour. The method is evaluated on artificial data. Results on real medical images show that the method is very effective in accurately locating object boundaries in very complex images.

Accurately segmenting and quantifying structures is a key issue in biomedical image analysis. The two conventional methods of image segmentation, region-based segmentation, and boundary finding, often suffer from a variety of limitations. Here the authors propose a method which endeavors to integrate the two approaches in an effort to form a unified approach that is robust to noise and poor initialization. The authors' approach uses Green's theorem to derive the boundary of a homogeneous region-classified area in the image and integrates this with a gray level gradient-based boundary finder. This combines the perceptual notions of edge/shape information with gray level homogeneity. A number of experiments were performed both on synthetic and real medical images of the brain and heart to evaluate the new approach, and it is shown that the integrated method typically performs better when compared to conventional gradient-based deformable boundary finding. Further, this method yields these improvements with little increase in computational overhead, an advantage derived from the application of the Green's theorem.

The authors propose a new approach for tracking the deformation of the left-ventricular (LV) myocardium from two-dimensional (2-D) magnetic resonance (MR) phase contrast velocity fields. The use of phase contrast MR velocity data in cardiac motion problems has been introduced by others (N.J. Pelc et al., 1991) and shown to be potentially useful for tracking discrete tissue elements, and therefore, characterizing LV motion. However, the authors show here that these velocity data: 1) are extremely noisy near the LV borders; and 2) cannot alone be used to estimate the motion and the deformation of the entire myocardium due to noise in the velocity fields. In this new approach, the authors use the natural spatial constraints of the endocardial and epicardial contours, detected semiautomatically in each image frame, to help remove noisy velocity vectors at the LV contours. The information from both the boundaries and the phase contrast velocity data is then integrated into a deforming mesh that is placed over the myocardium at one time frame and then tracked over the entire cardiac cycle. The deformation is guided by a Kalman filter that provides a compromise between 1) believing the dense field velocity and the contour data when it is crisp and coherent in a local spatial and temporal sense and 2) employing a temporally smooth cyclic model of cardiac motion when contour and velocity data are not trustworthy. The Kalman filter is particularly well suited to this task as it produces an optimal estimate of the left ventricle's kinematics (in the sense that the error is statistically minimized) given incomplete and noise corrupted data, and given a basic dynamical model of the left ventricle. The method has been evaluated with simulated data; the average error between tracked nodes and theoretical position was 1.8% of the total path length. The algorithm has also been evaluated with phantom data; the average error was 4.4% of the total path length. The authors show that in their initial tests with phantoms that the new approach shows small, but concrete improvements over previous techniques that used primarily phase contrast velocity data alone. They feel that these improvements will be amplified greatly as they move to direct comparisons in in vivo and three-dimensional (3-D) datasets.

Describes a new global shape parameterization for smoothly deformable three-dimensional (3-D) objects, such as those found in biomedical images, whose diversity and irregularity make them difficult to represent in terms of fixed features or parts. This representation is used for geometric surface matching to 3-D medical image data, such as from magnetic resonance imaging (MRI). The parameterization decomposes the surface into sinusoidal basis functions. Four types of surfaces are modeled: tori, open surfaces, closed surfaces and tubes. This parameterization allows a wide variety of smooth surfaces to be described with a small number of parameters. Extrinsic model-based information is incorporated by introducing prior probabilities on the parameters. Surface finding is formulated as an optimization problem. Results of the method applied to synthetic images and 3-D medical images of the heart and brain are presented.

Precise segmentation of underlying objects in an image is very important especially for biomedical image analysis. We present an integrated approach for boundary finding using region and curvature information along with the gradient. Unlike the previous methods, where smoothing is enforced by penalizing curvature, here the grey level curvature is used as an extra source of information. However, information fusion may not be useful unless used properly. To address that, we present results that highlight the pros and cons of using the various sources of information and indicate when one should get precedence over the others.>

A framework for tracking pointwise periodic non-rigid motion of the heart's left ventricular (LV) wall is presented which incorporates information from two different magnetic resonance imaging (MRI) techniques. New developments in phase-contrast cine MR imaging have produced spatial maps of instantaneous velocity that heave proven accuracy within the myocardium, or wall, of the heart. This information is combined with shape-based matching techniques to provide improved estimates of trajectories, especially in regions where shape information is limited. These raw trajectories act as input to a recursive least squares (RLS) filter which applies the constraints of temporal periodicity and spatial smoothness for the final estimate. The results of the RLS filter are compared with the motion of actual implanted markers. Comparisons are also made between exclusively shape-based filtered and phase-contrast enhanced trajectory estimates using both phantom and actual canine heart MR images.>

Accurate estimation of heart wall dense field motion and deformation could help to better understand the physiological processes associated with ischemic heart diseases, and to provide significant improvement in patient treatment. We present a new method of estimating left ventricular deformation which integrates instantaneous velocity information obtained within the mid-wall region with shape information found on the boundaries of the left ventricle. Velocity information is obtained from phase contrast magnetic resonance images, and boundary information is obtained from shape-based motion tracking of the endo- and cardial boundaries. The integration takes place within a continuum biomechanical heart model which is embedded in a finite element framework. We also employ a feedback mechanism to improve tracking accuracy. The integration of the two disparate but complementary sources overcomes some of the limitations of previous work in the field which concentrates on motion estimation from a single image-derived source.>

A framework for temporal analysis of left ventricular (LV) endocardial wall motion is presented. This approach uses harmonic estimation to model the periodic nature of cardiac motion. A method for flow vector computation is presented which defines a relationship between image-derived, shape-based correspondences and a more desirable, smoothly varying, set of correspondences. A recursive filter is then constructed which takes into consideration this relationship as well as knowledge of temporal trends. The trajectories of magnetic resonance (MR) contrast markers implanted in the LV wall are compared with trajectories estimated by the filter. The results of this comparison confirm that although cardiac motion is a 3-D problem, 2-D analysis provides a rich testing ground for algorithm development.

Medical tomographic images are formed by the intersection of the image plane and an object. As the image plane changes, different parts of the object come in view or drop out of view. However, for small changes of the image plane, most parts continue to remain visible and their qualitative embedding in the image remains similar. Therefore, similarity of part embeddings can be used to infer similarity of image planes. Part embeddings are useful features for other vision applications as well. In view of this, a spatial relation called "arrangement" is proposed to describe part embeddings. The relation describes how each part is surrounded by its neighbors. Further, a metric for arrangements is formulated by expressing arrangements in terms of the Voronoi diagram of the parts. Arrangements and their metric are used to retrieve images by image plane similarity in a cardiac magnetic resonance image database. Experiments with the database are reported which (1) validate the observation that similarity of image planes can be inferred from similarity of part embeddings, and (2) compare the performance of arrangement based image retrieval with that of expert radiologists.>

Accurately segmenting and quantifying structures is a key issue in biomedical image analysis. The two conventional methods of image segmentation, region-based segmentation and boundary finding, often suffer from a variety of limitations. We propose a method which endeavors to integrate the two approaches in an effort to form a unified approach that is robust to noise and poor initialization. Our approach uses Green's theorem to derive the boundary of a homogeneous region-classified area in the image and integrates this with a grey-level-gradient-based boundary finder. This combines the perceptual notions of edge/shape information with gray level homogeneity.>

This paper offers a new approach to integration of modules in an intelligent sensor system. Such a system requires that a set of modules-each doing a smaller portion of the overall task-be integrated within a unifying framework. From the perspective of computational systems, this problem holds a considerable interest because it is characterized by a set of coexisting mathematical objectives that need to be optimized simultaneously. In this sense, the design considerations necessitate the introduction of problem solving with multiple objectives. This paper explores these issues in the instance when each module is associated with a mathematical objective that is a function of the outputs of other modules. The integration problem is formulated and what is required of a good solution is presented. This examination interprets the decentralized mediation of conflicting subgoals as promoting a N-player game amongst the modules to be integrated and proposes a game-theoretic integration framework. We model the interaction among the modules as a noncooperative game and argue that this strategy leads to a framework in which the solutions correspond to a compromise decision. The application of this framework in image analysis motivates the hope that a framework such as game-theoretic integration will facilitate the development of general design principles for "modular" systems.>

The problem of tracking and quantifying the nonrigid nonuniform motion of naturally occurring annuli from image sequences is addressed. Motion computation is performed in two different manners, i.e., shape adherence with second order difference smoothing, and shape adherence with smoothing based on strain energy minimization. Results are shown for two typical applications, i.e., motion of the left ventricular wall of the heart over an entire cardiac cycle as derived from magnetic resonance images, and extension of neuronal growth cone membranes (lamel-lipodia) from light microscopy images.>

A qualitative spatial relation called 'arrangement' is proposed. Given an image, the relation describes the sequence in which neighbors of each part are situated around it. The relation captures a perceptual gestalt of the image. An application of the relation to similarity retrieval in medical image databases is discussed. 'Arrangement' is closely related to directional relations and extends the notion of directional relations to parts that have complex shapes. A metric for comparing 'arrangements' is also proposed. The metric is obtained by interpreting 'arrangements' in terms of Voronoi diagrams. The metric enables robust use of the relation in real-world situations.>

The problem of reinforcing local evidence of linear structure while suppressing unwanted information in noisy images is considered, using a modified form of relaxation labeling. The methodology is based on parametrizing a continuous set of orientation labels via a single vector and using a sigmoidal thresholding function to bias neighborhood influence and ensure convergence to a meaningful stable state. Label strength and label/no-label decisions are incorporated into a single functional. Optimal points of the functional represent the cases where as many pixels (objects) as possible have achieved the desirable linear-structure-reinforced and noise-suppressed labelings. Three different linear structure reinforcement tasks are considered within the general framework: edge reinforcement, edge reinforcement with thinning, and bar (line segment) reinforcement. Results from several image data sets are presented. This approach can directly handle continuous feature information from low-level image analysis operators, and the computational complexity of labeling is reduced.>

Segmentation using boundary finding is enhanced both by considering the boundary as a whole and by using model-based global shape information. The authors apply flexible constraints, in the form of a probabilistic deformable model, to the problem of segmenting natural 2-D objects whose diversity and irregularity of shape make them poorly represented in terms of fixed features or form. The parametric model is based on the elliptic Fourier decomposition of the boundary. Probability distributions on the parameters of the representation bias the model to a particular overall shape while allowing for deformations. Boundary finding is formulated as an optimization problem using a maximum a posteriori objective function. Results of the method applied to real and synthetic images are presented, including an evaluation of the dependence of the method on prior information and image quality.>

A variety of architectures have been proposed for integrating vision modules into a complete system. An examination of these systems indicates that they either lack analytic/computational tractability or are based on restrictive assumptions such as global additivity or sequential, decoupled solution of objectives. The authors introduce a framework of integration which overcomes both of these shortcomings. This proposed model is based on game-theoretic setting in which a correspondence between a vision system and a N-player game is established. The authors demonstrate the power of such a model within a comparative study in the context of a system aimed at delineating 2-D deformable contours from noisy images.>

The problem of measuring the motion of deformable objects from image sequences is addressed. The approach is based upon modeling the overall boundary of the object as a deformable contour and then tracking local segments of the contour through the temporal sequence. Motion computation involves first matching the local segments between pairs of contours by minimizing the deformation between the segments using a measure of bending energy. Results from the match process are incorporated into an optimization functional, along with a general smoothness term, whose local minimum results in a smooth flow field that is consistent with the match data. The computation is performed for all pairs of frames in the temporal sequence, resulting in a composite flow field over the entire sequence. The technique is applied to synthetic contour sequences and the problem of tracking left ventricular (LV) endocardial motion from medical image sequences.>

Describes a novel integration architecture for decentralized decision making in modular intelligent sensor systems. In contrast to previous approaches, this framework preserves the decentralized and coexisting natures of the objectives and is analytically and computationally tractable. The starting point for this approach is noncooperative N-player game theory.>

The problem of reinforcing local evidence of edges while suppressing unwanted information in noisy images is considered using a form of relaxation labeling. The methodology is based on parameterizing a continuous set of edge orientation labels using a single vector. A sigmoidal thresholding function similar to that used in artificial neural networks to bias neighborhood-influence and insure convergence to meaningful stable states is also utilized. A global optimization function is defined, and a decentralized parallel algorithm is derived that uses a steepest-gradient-descent approach to arrive at the optimal point on the functional surface, corresponding to desirable edge-reinforced and noise-suppressed labelings. In addition, a modification to the functional is presented which incorporates a thinning operation to insure that each edge is marked by only a single-pixel-wide response. Results from several image data sets indicate that the algorithm performs as well as or better than other relaxation labeling methods, and with improved computational efficiency.>

Segmentation using boundary finding is enhanced both by considering the boundary as a whole and by using model-based shape information. Flexible constraints, in the form of a probabilistic deformable model, are applied to the problem of segmenting natural objects whose diversity and irregularity of shape makes them poorly represented in terms of fixed features of forms. The parametric model is based on the elliptic Fourier decomposition of the boundary. The segmentation problem is solved as an optimization problem, where the best match between the boundary (as defined by the parameter vector) and the image data is found. Initial experimentation shows good results on a variety of images.>

In relaxation labeling, the domain model is incorporated into the system through the constraint functions ( compatibility coefficients). The selection of these functions is shown to be critical. In particular, for certain choices of the contraint functions, some processes will have a single nontrivial convergence point regardless of the initial labeling. Criteria on the constraint functions which result in a single convergence point are presented for those relaxation labeling processes whose formulations are within the framework of an optimization problem. We then present the necessary and sufficient conditions on the constraint €unctions under which the first of these sufficiency criteria is satisfied. The othpr conditions can be tested using standard techniques. These criteria yield a formal guideline for detecting some of the inadmi-ible constraint functions.