• A new method, UNISELF, is proposed to improve multiple sclerosis lesion segmentation. • UNISELF uses self-ensembled lesion fusion to improve accuracy and generalization. • UNISELF uses test-time instance normalization to address latent feature distribution shift. • UNISELF is among the top methods in the ISBI lesion segmentation challenge. • UNISELF outperforms other benchmarks on various out-of-domain test datasets. Automated segmentation of multiple sclerosis (MS) lesions using multicontrast magnetic resonance (MR) images improves efficiency and reproducibility compared to manual delineation, with deep learning (DL) methods achieving state-of-the-art performance. However, these DL-based methods have yet to simultaneously optimize in-domain accuracy and out-of-domain generalization when trained on a single source with limited data, or their performance has been unsatisfactory. To fill this gap, we propose a method called UNISELF, which achieves high accuracy within a single training domain while demonstrating strong generalizability across multiple out-of-domain test datasets. UNISELF employs a novel test-time self-ensembled lesion fusion to improve segmentation accuracy, and leverages test-time instance normalization (TTIN) of latent features to address domain shifts and missing input contrasts. Trained on the ISBI 2015 longitudinal MS segmentation challenge training dataset, UNISELF ranks among the best-performing methods on the challenge test dataset. Additionally, UNISELF outperforms all benchmark methods trained on the same ISBI training data across diverse out-of-domain test datasets with domain shifts and missing contrasts, including the public MICCAI 2016 and UMCL datasets, as well as a private multisite dataset. These test datasets exhibit domain shifts and/or missing contrasts caused by variations in acquisition protocols, scanner types, and imaging artifacts arising from imperfect acquisition. Our code is available at https://github.com/Jinwei1209/UNISELF .

Tagged magnetic resonance imaging (tMRI) is a valuable tool for visualizing and quantifying tissue deformation in vivo. Its use is often hampered, however, by tag fading, long computation times, and the challenge of ensuring diffeomorphic, incompressible motion fields. In this paper, we describe a novel integration of the harmonic phase (HARP) approach to tMRI analysis with an unsupervised deep learning-based registration framework to estimate 2D and 3D motion fields that are diffeomorphic and nearly incompressible. The resulting method, called deep sinusoidally transformed HARP, or DSHARP, enables end-to-end network training by implementing a transformation of the harmonic phase to remove phase-wrapping discontinuities. It produces diffeomorphic motion by estimating a stationary velocity field from which motion is computed using the scaling and squaring technique. Finally, it encourages incompressibility using a novel Jacobian determinant loss term during network training. We evaluated DSHARP on 2D and 3D phantom data with simulated incompressible motions, real 3D human tongue data acquired during speech from both healthy and glossectomy subjects, and cardiac tagged MRI from the public STACOM 2011 benchmark. Our approach outperforms HARP, SinMod, SyN, PVIRA, VoxelMorph, and DeepTag in tracking accuracy, computation speed, and preservation of incompressibility.

Medical image harmonization aims to reduce the differences in appearance caused by scanner hardware variations to allow for consistent and reliable comparisons across devices. Harmonization based on paired images from different devices has limited applicability in real-world clinical settings. On the other hand, unpaired harmonization typically does not guarantee anatomy consistency, which is problematic because anatomical information preservation is paramount. The Schrödinger bridge framework has achieved state-of-the-art style transfer performance with natural images by matching distributions of unpaired images, but this approach can also introduce anatomy changes when applied to medical images. We show that such changes occur because the Schrödinger bridge uses the square of the Euclidean distance between images as the transport cost in an entropy-regularized optimal transport problem. Such a transport cost is not appropriate for measuring anatomical distances, as medical images with the same anatomy need not have a small Euclidean distance between them. In this paper, we propose a latent metric Schrödinger bridge (LMSB) framework to improve the anatomical consistency for the harmonization of medical images. We develop an invertible network that maps medical images into a latent Euclidean metric space where the distances among images with the same anatomy are minimized using the pullback latent metric. Within this latent space, we train a Schrödinger bridge to match distributions. We show that the proposed LMSB is superior to the direct application of a Schrödinger bridge to harmonize optical coherence tomography (OCT) images.

Magnetic Resonance Imaging (MRI) allows analyzing speech production by capturing high-resolution images of the dynamic processes in the vocal tract. In clinical applications, combining MRI with synchronized speech recordings leads to improved patient outcomes, especially if a phonological-based approach is used for assessment. However, when audio signals are unavailable, the recognition accuracy of sounds is decreased when using only MRI data. We propose a contrastive learning approach to improve the detection of phonological classes from MRI data when acoustic signals are not available at inference time. We demonstrate that frame-wise recognition of phonological classes improves from an f1 of 0.74 to 0.85 when the contrastive loss approach is implemented. Furthermore, we show the utility of our approach in the clinical application of using such phonological classes to assess speech disorders in patients with tongue cancer, yielding promising results in the recognition task.

Abstract Producing spatial transformations that are diffeomorphic is a key goal in deformable image registration. As a diffeomorphic transformation should have positive Jacobian determinant $$\vert J\vert $$ | J | everywhere, the number of pixels (2D) or voxels (3D) with $$\vert J\vert <0$$ | J | < 0 has been used to test for diffeomorphism and also to measure the irregularity of the transformation. For digital transformations, $$\vert J\vert $$ | J | is commonly approximated using a central difference, but this strategy can yield positive $$\vert J\vert $$ | J | ’s for transformations that are clearly not diffeomorphic—even at the pixel or voxel resolution level. To show this, we first investigate the geometric meaning of different finite difference approximations of $$\vert J\vert $$ | J | . We show that to determine if a deformation is diffeomorphic for digital images, the use of any individual finite difference approximation of $$\vert J\vert $$ | J | is insufficient. We further demonstrate that for a 2D transformation, four unique finite difference approximations of $$\vert J\vert $$ | J | ’s must be positive to ensure that the entire domain is invertible and free of folding at the pixel level. For a 3D transformation, ten unique finite differences approximations of $$\vert J\vert $$ | J | ’s are required to be positive. Our proposed digital diffeomorphism criteria solves several errors inherent in the central difference approximation of $$\vert J\vert $$ | J | and accurately detects non-diffeomorphic digital transformations. The source code of this work is available at https://github.com/yihao6/digital_diffeomorphism .

Finite element models (FEM) of the tongue have facilitated speech studies through analysis of internal muscle forces indirectly derived from imaging data. In this work, we build a uniform hexahedral FEM of a tongue atlas constructed from magnetic resonance imaging data of a healthy population. The FEM is driven by inverse internal tongue tissue kinematics of speakers temporally aligned and deformed into the same atlas space, while performing the speech task "a souk" allowing muscle activation predictions. This work aims to investigate the commonalities in tongue motor strategies in the articulation of "a souk" predicted by the inverse tongue atlas model. Our findings report variability among five speakers for estimated muscle activations with a similarity index using a dynamic time warp function. Two speakers show similarity index > 0.9 and two others < 0.7 with respect to a reference speaker for most tongue muscles. The relative motion tracking error of the model is less than 2% which is promising for speech study applications.

Multimodal representation learning using visual movements from cine magnetic resonance imaging (MRI) and their acoustics has shown great potential to learn shared representation and to predict one modality from another. Here, we propose a new synthesis framework to translate from cine MRI sequences to spectrograms with a limited dataset size. Our framework hinges on a novel fully convolutional heterogeneous translator, with a 3D CNN encoder for efficient sequence encoding and a 2D transpose convolution decoder. In addition, a pairwise correlation of the samples with the same speech word is utilized with a latent space representation disentanglement scheme. Furthermore, an adversarial training approach with generative adversarial networks is incorporated to provide enhanced realism on our generated spectrograms. Our experimental results, carried out with a total of 63 cine MRI sequences alongside speech acoustics, show that our framework improves synthesis accuracy, compared with competing methods. Our framework thereby has shown the potential to aid in better understanding the relationship between the two modalities.

Optical coherence tomography angiography (OCTA) is an imaging modality that can be used for analyzing retinal vasculature. Quantitative assessment of en face OCTA images requires accurate segmentation of the capillaries. Using deep learning approaches for this task faces two major challenges. First, acquiring sufficient manual delineations for training can take hundreds of hours. Second, OCTA images suffer from numerous contrast-related artifacts that are currently inherent to the modality and vary dramatically across scanners. We propose to solve both problems by learning a disentanglement of an anatomy component and a local contrast component from paired OCTA scans. With the contrast removed from the anatomy component, a deep learning model that takes the anatomy component as input can learn to segment vessels with a limited portion of the training images being manually labeled. Our method demonstrates state-of-the-art performance for OCTA vessel segmentation.

Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in deep learning are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, etc. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.

High resolution magnetic resonance (MR) images are desired in many clinical and research applications. Acquiring such images with high signal-to-noise (SNR), however, can require a long scan duration, which is difficult for patient comfort, is more costly, and makes the images susceptible to motion artifacts. A very common practical compromise for both 2D and 3D MR imaging protocols is to acquire volumetric MR images with high in-plane resolution, but lower through-plane resolution. In addition to having poor resolution in one orientation, 2D MRI acquisitions will also have aliasing artifacts, which further degrade the appearance of these images. This paper presents an approach SMORE1 based on convolutional neural networks (CNNs) that restores image quality by improving resolution and reducing aliasing in MR images.2 This approach is self-supervised, which requires no external training data because the high-resolution and low-resolution data that are present in the image itself are used for training. For 3D MRI, the method consists of only one self-supervised super-resolution (SSR) deep CNN that is trained from the volumetric image data. For 2D MRI, there is a self-supervised anti-aliasing (SAA) deep CNN that precedes the SSR CNN, also trained from the volumetric image data. Both methods were evaluated on a broad collection of MR data, including filtered and downsampled images so that quantitative metrics could be computed and compared, and actual acquired low resolution images for which visual and sharpness measures could be computed and compared. The super-resolution method is shown to be visually and quantitatively superior to previously reported methods.

Synthesizing a CT image from an available MR image has recently emerged as a key goal in radiotherapy treatment planning for cancer patients. CycleGANs have achieved promising results on unsupervised MR-to-CT image synthesis; however, because they have no direct constraints between input and synthetic images, cycleGANs do not guarantee structural consistency between these two images. This means that anatomical geometry can be shifted in the synthetic CT images, clearly a highly undesirable outcome in the given application. In this paper, we propose a structure-constrained cycleGAN for unsupervised MR-to-CT synthesis by defining an extra structure-consistency loss based on the modality independent neighborhood descriptor. We also utilize a spectral normalization technique to stabilize the training process and a self-attention module to model the long-range spatial dependencies in the synthetic images. Results on unpaired brain and abdomen MR-to-CT image synthesis show that our method produces better synthetic CT images in both accuracy and visual quality as compared to other unsupervised synthesis methods. We also show that an approximate affine pre-registration for unpaired training data can improve synthesis results.

Muscle coordination patterns of lingual behaviors are synergies generated by deforming local muscle groups in a variety of ways. Functional units are functional muscle groups of local structural elements within the tongue that compress, expand, and move in a cohesive and consistent manner. Identifying the functional units using tagged-magnetic resonance imaging (MRI) sheds light on the mechanisms of normal and pathological muscle coordination patterns, yielding improvement in surgical planning, treatment, or rehabilitation procedures. In this paper, to mine this information, we propose a matrix factorization and probabilistic graphical model framework to produce building blocks and their associated weighting map using motion quantities extracted from tagged-MRI. Our tagged-MRI imaging and accurate voxel-level tracking provide previously unavailable internal tongue motion patterns, thus revealing the inner workings of the tongue during speech or other lingual behaviors. We then employ spectral clustering on the weighting map to identify the cohesive regions defined by the tongue motion that may involve multiple or undocumented regions. To evaluate our method, we perform a series of experiments. We first use two-dimensional images and synthetic data to demonstrate the accuracy of our method. We then use three-dimensional synthetic and in vivo tongue motion data using protrusion and simple speech tasks to identify subject-specific and data-driven functional units of the tongue in localized regions.

This editorial introduces the Special Issue on Simulation and Synthesis in Medical Imaging. In this editorial, we define so-far ambiguous terms of simulation and synthesis in medical imaging. We also briefly discuss the synergistic importance of mechanistic (hypothesis-driven) and phenomenological (data-driven) models of medical image generation. Finally, we introduce the twelve papers published in this issue covering both mechanistic (5) and phenomenological (7) medical image generation. This rich selection of papers covers applications in cardiology, retinopathy, histopathology, neurosciences, and oncology. It also covers all mainstream diagnostic medical imaging modalities. We conclude the editorial with a personal view on the field and highlight some existing challenges and future research opportunities.

Tagged magnetic resonance imaging has been used for decades to observe and quantify motion and strain of deforming tissue. It is challenging to obtain 3-D motion estimates due to a tradeoff between image slice density and acquisition time. Typically, interpolation methods are used either to combine 2-D motion extracted from sparse slice acquisitions into 3-D motion or to construct a dense volume from sparse acquisitions before image registration methods are applied. This paper proposes a new phase-based 3-D motion estimation technique that first computes harmonic phase volumes from interpolated tagged slices and then matches them using an image registration framework. The approach uses several concepts from diffeomorphic image registration with a key novelty that defines a symmetric similarity metric on harmonic phase volumes from multiple orientations. The material property of harmonic phase solves the aperture problem of optical flow and intensity-based methods and is robust to tag fading. A harmonic magnitude volume is used in enforcing incompressibility in the tissue regions. The estimated motion fields are dense, incompressible, diffeomorphic, and inverse-consistent at a 3-D voxel level. The method was evaluated using simulated phantoms, human brain data in mild head accelerations, human tongue data during speech, and an open cardiac data set. The method shows comparable accuracy to three existing methods while demonstrating low computation time and robustness to tag fading and noise.

It is faster and therefore cheaper to acquire magnetic resonance images (MRI) with higher in-plane resolution than through-plane resolution. The low resolution of such acquisitions can be increased using post-processing techniques referred to as super-resolution (SR) algorithms. SR is known to be an ill-posed problem. Most state-of-the-art SR algorithms rely on the presence of external/training data to learn a transform that converts low resolution input to a higher resolution output. In this paper an SR approach is presented that is not dependent on any external training data and is only reliant on the acquired image. Patches extracted from the acquired image are used to estimate a set of new images, where each image has increased resolution along a particular direction. The final SR image is estimated by combining images in this set via the technique of Fourier Burst Accumulation. Our approach was validated on simulated low resolution MRI images, and showed significant improvement in image quality and segmentation accuracy when compared to competing SR methods. SR of FLuid Attenuated Inversion Recovery (FLAIR) images with lesions is also demonstrated.

Intraoperative localization of target anatomy and critical structures defined in preoperative MR/CT images can be achieved through the use of multimodality deformable registration. We propose a symmetric diffeomorphic deformable registration algorithm incorporating a modality-independent neighborhood descriptor (MIND) and a robust Huber metric for MR-to-CT registration. The method, called MIND Demons, finds a deformation field between two images by optimizing an energy functional that incorporates both the forward and inverse deformations, smoothness on the integrated velocity fields, a modality-insensitive similarity function suitable to multimodality images, and smoothness on the diffeomorphisms themselves. Direct optimization without relying on the exponential map and stationary velocity field approximation used in conventional diffeomorphic Demons is carried out using a Gauss-Newton method for fast convergence. Registration performance and sensitivity to registration parameters were analyzed in simulation, phantom experiments, and clinical studies emulating application in image-guided spine surgery, and results were compared to mutual information (MI) free-form deformation (FFD), local MI (LMI) FFD, normalized MI (NMI) Demons, and MIND with a diffusion-based registration method (MIND-elastic). The method yielded sub-voxel invertibility (0.008 mm) and nonzero-positive Jacobian determinants. It also showed improved registration accuracy in comparison to the reference methods, with mean target registration error (TRE) of 1.7 mm compared to 11.3, 3.1, 5.6, and 2.4 mm for MI FFD, LMI FFD, NMI Demons, and MIND-elastic methods, respectively. Validation in clinical studies demonstrated realistic deformations with sub-voxel TRE in cases of cervical, thoracic, and lumbar spine.

Multimodal image registration is a class of algorithms to find correspondence from different modalities. Since different modalities do not exhibit the same characteristics, finding accurate correspondence still remains a challenge. To deal with this, mutual information (MI)-based registration has been a preferred choice as MI is based on the statistical relationship between both volumes to be registered. However, MI has some limitations. First, MI-based registration often fails when there are local intensity variations in the volumes. Second, MI only considers the statistical intensity relationships between both volumes and ignores the spatial and geometric information about the voxel. In this work, we propose to address these limitations by incorporating spatial and geometric information via a 3D Harris operator. In particular, we focus on the registration between a high-resolution image and a low-resolution image. The MI cost function is computed in the regions where there are large spatial variations such as corner or edge. In addition, the MI cost function is augmented with geometric information derived from the 3D Harris operator applied to the high-resolution image. The robustness and accuracy of the proposed method were demonstrated using experiments on synthetic and clinical data including the brain and the tongue. The proposed method provided accurate registration and yielded better performance over standard registration methods.

Quantitative measurements from segmentations of human brain magnetic resonance (MR) images provide important biomarkers for normal aging and disease progression. In this paper, we propose a patch-based tissue classification method from MR images that uses a sparse dictionary learning approach and atlas priors. Training data for the method consists of an atlas MR image, prior information maps depicting where different tissues are expected to be located, and a hard segmentation. Unlike most atlas-based classification methods that require deformable registration of the atlas priors to the subject, only affine registration is required between the subject and training atlas. A subject-specific patch dictionary is created by learning relevant patches from the atlas. Then the subject patches are modeled as sparse combinations of learned atlas patches leading to tissue memberships at each voxel. The combination of prior information in an example-based framework enables us to distinguish tissues having similar intensities but different spatial locations. We demonstrate the efficacy of the approach on the application of whole-brain tissue segmentation in subjects with healthy anatomy and normal pressure hydrocephalus, as well as lesion segmentation in multiple sclerosis patients. For each application, quantitative comparisons are made against publicly available state-of-the art approaches.

The performance of image analysis algorithms applied to magnetic resonance images is strongly influenced by the pulse sequences used to acquire the images. Algorithms are typically optimized for a targeted tissue contrast obtained from a particular implementation of a pulse sequence on a specific scanner. There are many practical situations, including multi-institution trials, rapid emergency scans, and scientific use of historical data, where the images are not acquired according to an optimal protocol or the desired tissue contrast is entirely missing. This paper introduces an image restoration technique that recovers images with both the desired tissue contrast and a normalized intensity profile. This is done using patches in the acquired images and an atlas containing patches of the acquired and desired tissue contrasts. The method is an example-based approach relying on sparse reconstruction from image patches. Its performance in demonstrated using several examples, including image intensity normalization, missing tissue contrast recovery, automatic segmentation, and multimodal registration. These examples demonstrate potential practical uses and also illustrate limitations of our approach.

Image labeling and parcellation (i.e., assigning structure to a collection of voxels) are critical tasks for the assessment of volumetric and morphometric features in medical imaging data. The process of image labeling is inherently error prone as images are corrupted by noise and artifacts. Even expert interpretations are subject to subjectivity and the precision of the individual raters. Hence, all labels must be considered imperfect with some degree of inherent variability. One may seek multiple independent assessments to both reduce this variability and quantify the degree of uncertainty. Existing techniques have exploited maximum a posteriori statistics to combine data from multiple raters and simultaneously estimate rater reliabilities. Although quite successful, wide-scale application has been hampered by unstable estimation with practical datasets, for example, with label sets with small or thin objects to be labeled or with partial or limited datasets. As well, these approaches have required each rater to generate a complete dataset, which is often impossible given both human foibles and the typical turnover rate of raters in a research or clinical environment. Herein, we propose a robust approach to improve estimation performance with small anatomical structures, allow for missing data, account for repeated label sets, and utilize training/catch trial data. With this approach, numerous raters can label small, overlapping portions of a large dataset, and rater heterogeneity can be robustly controlled while simultaneously estimating a single, reliable label set and characterizing uncertainty. The proposed approach enables many individuals to collaborate in the construction of large datasets for labeling tasks (e.g., human parallel processing) and reduces the otherwise detrimental impact of rater unavailability.

Measuring the 3D motion of muscular tissues, e.g., the heart or the tongue, using magnetic resonance (MR) tagging is typically carried out by interpolating the 2D motion information measured on orthogonal stacks of images. The incompressibility of muscle tissue is an important constraint on the reconstructed motion field and can significantly help to counter the sparsity and incompleteness of the available motion information. Previous methods utilizing this fact produced incompressible motions with limited accuracy. In this paper, we present an incompressible deformation estimation algorithm (IDEA) that reconstructs a dense representation of the 3D displacement field from tagged MR images and the estimated motion field is incompressible to high precision. At each imaged time frame, the tagged images are first processed to determine components of the displacement vector at each pixel relative to the reference time. IDEA then applies a smoothing, divergence-free, vector spline to interpolate velocity fields at intermediate discrete times such that the collection of velocity fields integrate over time to match the observed displacement components. Through this process, IDEA yields a dense estimate of a 3D displacement field that matches our observations and also corresponds to an incompressible motion. The method was validated with both numerical simulation and in vivo human experiments on the heart and the tongue.

The likelihood of finding manufactured components (surgical tools, implants, etc.) within a tomographic field-of-view has been steadily increasing. One reason is the aging population and proliferation of prosthetic devices, such that more people undergoing diagnostic imaging have existing implants, particularly hip and knee implants. Another reason is that use of intraoperative imaging (e.g., cone-beam CT) for surgical guidance is increasing, wherein surgical tools and devices such as screws and plates are placed within or near to the target anatomy. When these components contain metal, the reconstructed volumes are likely to contain severe artifacts that adversely affect the image quality in tissues both near and far from the component. Because physical models of such components exist, there is a unique opportunity to integrate this knowledge into the reconstruction algorithm to reduce these artifacts. We present a model-based penalized-likelihood estimation approach that explicitly incorporates known information about component geometry and composition. The approach uses an alternating maximization method that jointly estimates the anatomy and the position and pose of each of the known components. We demonstrate that the proposed method can produce nearly artifact-free images even near the boundary of a metal implant in simulated vertebral pedicle screw reconstructions and even under conditions of substantial photon starvation. The simultaneous estimation of device pose also provides quantitative information on device placement that could be valuable to quality assurance and verification of treatment delivery.

The success of prostate brachytherapy critically depends on delivering adequate dose to the prostate gland. Intraoperative localization of the implanted seeds provides potential for dose evaluation and optimization during therapy. A reduced-dimensionality matching algorithm for prostate brachytherapy seed reconstruction (REDMAPS) that uses multiple X-ray fluoroscopy images obtained from different poses is proposed. The seed reconstruction problem is formulated as a combinatorial optimization problem, and REDMAPS finds a solution in a clinically acceptable amount of time using dimensionality reduction to create a smaller space of possible solutions. Dimensionality reduction is possible since the optimal solution has approximately zero cost when the poses of the acquired images are known to be within a small error. REDMAPS is also formulated to address the "hidden seed problem" in which seeds overlap on one or more observed images. REDMAPS uses a pruning algorithm to avoid unnecessary computation of cost metrics and the reduced problem is solved using linear programming. REDMAPS was first evaluated and its parameters tuned using simulations. It was then validated using five phantom and 21 patient datasets. REDMAPS was successful in reconstructing the seeds with an overall seed matching rate above 99% and a reconstruction error below 1 mm in less than 5 s.

We propose a method for improving the quality of cone-beam tomographic reconstruction done with a C-arm. C-arm scans frequently suffer from incomplete information due to image truncation, limited scan length, or other limitations. Our proposed "hybrid reconstruction" method injects information from a prior anatomical model, derived from a subject-specific computed tomography (CT) or from a statistical database (atlas), where the C-arm X-ray data is missing. This significantly reduces reconstruction artifacts with little loss of true information from the X-ray projections. The methods consist of constructing anatomical models, fast rendering of digitally reconstructed radiograph (DRR) projections of the models, rigid or deformable registration of the model and the X-ray images, and fusion of the DRR and X-ray projections, all prior to a conventional filtered back-projection algorithm. Our experiments, conducted with a mobile image intensifier C-arm, demonstrate visually and quantitatively the contribution of data fusion to image quality, which we assess through comparison to a "ground truth" CT. Importantly, we show that a significantly improved reconstruction can be obtained from a C-arm scan as short as 90° by complementing the observed projections with DRRs of two prior models, namely an atlas and a preoperative same-patient CT. The hybrid reconstruction principles are applicable to other types of C-arms as well.

Magnetic resonance tagging makes it possible to measure the motion of tissues such as muscles in the heart and tongue. The harmonic phase (HARP) method largely automates the process of tracking points within tagged MR images, permitting many motion properties to be computed. However, HARP tracking can yield erroneous motion estimates due to 1) large deformations between image frames, 2) through-plane motion, and 3) tissue boundaries. Methods that incorporate the spatial continuity of motion--so-called refinement or flood-filling methods--have previously been reported to reduce tracking errors. This paper presents a new refinement method based on shortest path computations. The method uses a graph representation of the image and seeks an optimal tracking order from a specified seed to each point in the image by solving a single source shortest path problem. This minimizes the potential errors for those path dependent solutions that are found in other refinement methods. In addition to this, tracking in the presence of through-plane motion is improved by introducing synthetic tags at the reference time (when the tissue is not deformed). Experimental results on both tongue and cardiac images show that the proposed method can track the whole tissue more robustly and is also computationally efficient.

Intraoperative dosimetry in prostate brachytherapy requires localization of the implanted radioactive seeds. A tomosynthesis-based seed reconstruction method is proposed. A three-dimensional volume is reconstructed from Gaussian-blurred projection images and candidate seed locations are computed from the reconstructed volume. A false positive seed removal process, formulated as an optimal coverage problem, iteratively removes "ghost" seeds that are created by tomosynthesis reconstruction. In an effort to minimize pose errors that are common in conventional C-arms, initial pose parameter estimates are iteratively corrected by using the detected candidate seeds as fiducials, which automatically "focuses" the collected images and improves successive reconstructed volumes. Simulation results imply that the implanted seed locations can be estimated with a detection rate of > or = 97.9% and > or = 99.3% from three and four images, respectively, when the C-arm is calibrated and the pose of the C-arm is known. The algorithm was also validated on phantom data sets successfully localizing the implanted seeds from four or five images. In a Phase-1 clinical trial, we were able to localize the implanted seeds from five intraoperative fluoroscopy images with 98.8% (STD=1.6) overall detection rate.

The simultaneous segmentation of multiple objects is an important problem in many imaging and computer vision applications. Various extensions of level set segmentation techniques to multiple objects have been proposed; however, no one method maintains object relationships, preserves topology, is computationally efficient, and provides an object-dependent internal and external force capability. In this paper, a framework for segmenting multiple objects that permits different forces to be applied to different boundaries while maintaining object topology and relationships is presented. Because of this framework, the segmentation of multiple objects each with multiple compartments is supported, and no overlaps or vacuums are generated. The computational complexity of this approach is independent of the number of objects to segment, thereby permitting the simultaneous segmentation of a large number of components. The properties of this approach and comparisons to existing methods are shown using a variety of images, both synthetic and real.

Spatial normalization is frequently used to map data to a standard coordinate system by removing intersubject morphological differences, thereby allowing for group analysis to be carried out. The work presented in this paper is motivated by the need for an automated cortical surface normalization technique that will automatically identify homologous cortical landmarks and map them to the same coordinates on a standard manifold. The geometry of a cortical surface is analyzed using two shape measures that distinguish the sulcal and gyral regions in a multiscale framework. A multichannel optical flow warping procedure aligns these shape measures between a reference brain and a subject brain, creating the desired normalization. The partial differential equation that carries out the warping is implemented in a Euclidean framework in order to facilitate a multiresolution strategy, thereby permitting large deformations between the two surfaces. The technique is demonstrated by aligning 33 normal cortical surfaces and showing both improved structural alignment in manually labeled sulci and improved functional alignment in positron emission tomography data mapped to the surfaces. A quantitative comparison between our proposed surface-based spatial normalization method and a leading volumetric spatial normalization method is included to show that the surface-based spatial normalization performs better in matching homologous cortical anatomies.

Topology-preserving geometric deformable models (TGDMs) are used to segment objects that have a known topology. Their accuracy is inherently limited, however, by the resolution of the underlying computational grid. Although this can be overcome by using fine-resolution grids, both the computational cost and the size of the resulting contour increase dramatically. In order to maintain computational efficiency and to keep the contour size manageable, we have developed a new framework, termed QTGDMs, for topology-preserving geometric deformable models on balanced quadtree grids (BQGs). In order to do this, definitions and concepts from digital topology on regular grids were extended to BQGs so that characterization of simple points could be made. Other issues critical to the implementation of geometric deformable models are also addressed and a strategy for adapting a BQG during contour evolution is presented. We demonstrate the performance of the QTGDM method using both mathematical phantoms and real medical images.

Diffusion tensor imaging (DTI) is widely used to characterize white matter in health and disease. Previous approaches to the estimation of diffusion tensors have either been statistically suboptimal or have used Gaussian approximations of the underlying noise structure, which is Rician in reality. This can cause quantities derived from these tensors - e.g., fractional anisotropy and apparent diffusion coefficient - to diverge from their true values, potentially leading to artifactual changes that confound clinically significant ones. This paper presents a novel maximum likelihood approach to tensor estimation, denoted Diffusion Tensor Estimation by Maximizing Rician Likelihood (DTEMRL). In contrast to previous approaches, DTEMRL considers the joint distribution of all observed data in the context of an augmented tensor model to account for variable levels of Rician noise. To improve numeric stability and prevent non-physical solutions, DTEMRL incorporates a robust characterization of positive definite tensors and a new estimator of underlying noise variance. In simulated and clinical data, mean squared error metrics show consistent and significant improvements from low clinical SNR to high SNR. DTEMRL may be readily supplemented with spatial regularization or a priori tensor distributions for Bayesian tensor estimation.

We present a novel approach to efficiently compute thickness, correspondence, and gridding of tissues between two simply connected boundaries. The solution of Laplace's equation within the tissue region provides a harmonic function whose gradient flow determines the correspondence trajectories going from one boundary to the other. The proposed method uses and expands upon two recently introduced techniques in order to compute thickness and correspondences based on these trajectories. Pairs of partial differential equations are efficiently computed within an Eulerian framework and combined with a Lagrangian approach so that correspondences trajectories are partially constructed when necessary. Examples are presented in order to compare the performance of this method with those of the pure Lagrangian and pure Eulerian approaches. Results show that the proposed technique takes advantage of both the speed of the Eulerian approach and the accuracy of the Lagrangian approach.

Geometric deformable models based on the level set method have become very popular. To overcome an inherent limitation in accuracy while maintaining computational efficiency, adaptive grid techniques using local grid refinement have been developed for use with these models. However, this strategy requires a very complex data structure, yields large numbers of contour points, and is inconsistent with our previously presented topology-preserving geometric deformable model (TGDM). In this paper, we incorporate an alternative adaptive grid technique called the moving grid method into the geometric deformable model framework. We find that it is simpler to implement than grid refinement, requiring no large, complex, hierarchical data structures. It also limits the number of contour vertices in the final contour and supports the incorporation of the topology-preserving constraint of TGDM. After presenting the algorithm, we demonstrate its performance using both simulated and real images.

Active contour and surface models, also known as deformable models, are powerful image segmentation techniques. Geometric deformable models implemented using level set methods have advantages over parametric models due to their intrinsic behavior, parameterization independence, and ease of implementation. However, a long claimed advantage of geometric deformable models-the ability to automatically handle topology changes-turns out to be a liability in applications where the object to be segmented has a known topology that must be preserved. We present a new class of geometric deformable models designed using a novel topology-preserving level set method, which achieves topology preservation by applying the simple point concept from digital topology. These new models maintain the other advantages of standard geometric deformable models including subpixel accuracy and production of nonintersecting curves or surfaces. Moreover, since the topology-preserving constraint is enforced efficiently through local computations, the resulting algorithm incurs only nominal computational overhead over standard geometric deformable models. Several experiments on simulated and real data are provided to demonstrate the performance of this new deformable model algorithm.

We outline an Eulerian framework for computing the thickness of tissues between two simply connected boundaries that does not require landmark points or parameterizations of either boundary. Thickness is defined as the length of correspondence trajectories, which run from one tissue boundary to the other, and which follow a smooth vector field constructed in the region between the boundaries. A pair of partial differential equations (PDEs) that are guided by this vector field are then solved over this region, and the sum of their solutions yields the thickness of the tissue region. Unlike other approaches, this approach does not require explicit construction of any correspondence trajectories. An efficient, stable, and computationally fast solution to these PDEs is found by careful selection of finite differences according to an upwinding condition. The behavior and performance of our method is demonstrated on two simulations and two magnetic resonance imaging data sets in two and three dimensions. These experiments reveal very good performance and show strong potential for application in tissue thickness visualization and quantification.

Reconstructing an accurate and topologically correct representation of the cortical surface of the brain is an important objective in various neuroscience applications. Most cortical surface reconstruction methods either ignore topology or correct it using manual editing or methods that lead to inaccurate reconstructions. Shattuck and Leahy recently reported a fully automatic method that yields a topologically correct representation with little distortion of the underlying segmentation. We provide an alternate approach that has several advantages over their approach, including the use of arbitrary digital connectivities, a flexible morphology-based multiscale approach, and the option of foreground-only or background-only correction. A detailed analysis of the method's performance on 15 magnetic resonance brain images is provided.

A method for automated segmentation of major cortical sulci on the outer brain boundary is presented, with emphasis on automatically determining point correspondence and on labeling cortical regions. The method is formulated in a general optimization framework defined on the unit sphere, which serves as parametric domain for convoluted surfaces of spherical topology. A statistical shape model, which includes a network of deformable curves on the unit sphere, seeks geometric features such as high curvature regions and labels such features via a deformation process that is confined within a spherical map of the outer brain boundary. The limitations of the customary spherical coordinate system, which include discontinuities at the poles and nonuniform sampling, are overcome by defining the statistical prior of shape variation in terms of projections of landmark points onto corresponding tangent planes of the sphere. The method is tested against and shown to be as accurate as manually defined segmentations.

Active contour and surface models, also known as deformable models, constitute a class of powerful segmentation techniques. Geometric deformable models implemented via level-set methods have advantages over parametric ones due to their intrinsic behavior, parameterization independence, and ease of implementation. However, a long claimed advantage of geometric deformable models, the ability to automatically handle topology changes, turns out to be a liability in applications where the objects to be segmented have a known topology that must be preserved. In this paper, we present a geometric deformable model that preserves topology using the simple point concept from digital topology. This algorithm maintains the other advantages of standard geometric deformable models including sub-pixel accuracy and production of nonintersecting curves (or surfaces). Several experiments on simulated and real data are provided to demonstrate the performance of the proposed algorithm.

We outline an Eulerian framework for computing the thickness of tissues between two simply connected boundaries. Thickness is defined as the length of trajectories which follow a smooth vector field constructed in the region between the boundaries. A pair of partial differential equations (PDEs) are then solved and combined to yield length without requiring the explicit construction of the trajectories. An efficient, stable, and computationally fast solution to these PDEs is found by careful selection of finite differences according to an upwinding condition. The behavior and performance of the method is demonstrated on two simulations and two magnetic resonance imaging data sets in two and three dimensions. These experiments reveal very good performance and show strong potential for application in tissue thickness visualization and quantification.

In this paper we improve the extraction of harmonic images from tagged MR images of the heart. Using simulations, we study the shapes of the spectra of harmonic images and their relation to the heart motion. The filters used to extract the harmonic images are designed based on the shape of the spectra and tuned using simulations. The design also considers additive noise and interference from other harmonic images.

This paper describes a new image processing technique for rapid analysis and visualization of tagged cardiac magnetic resonance (MR) images. The method is based on the use of isolated spectral peaks in spatial modulation of magnetization (SPAMM)-tagged magnetic resonance images. We call the calculated angle of the complex image corresponding to one of these peaks a harmonic phase (HARP) image and show that HARP images can be used to synthesize conventional tag lines, reconstruct displacement fields for small motions, and calculate two-dimensional (2-D) strain. The performance of this new approach is demonstrated using both real and simulated tagged MR images. Potential for use of HARP images in fast imaging techniques and three-dimensional (3-D) analyses are discussed.

A general active contour formulation is considered and a convexity analysis of its energy function is presented. Conditions under which this formulation has a unique solution are derived; these conditions involve both the active contour energy potential and the regularization parameters. This analysis is then applied to four particular active contour formulations, revealing important characteristics about their convexity, and suggesting that external potentials involving center-of-mass computations may be better behaved than the usual potentials based on image gradients. Our analysis also provides an explanation for the poor convergence behavior at concave boundaries and suggests an alternate algorithm for approaching these types of boundaries.

We present a novel algorithm for obtaining fuzzy segmentations of images that are subject to multiplicative intensity inhomogeneities, such as magnetic resonance images. The algorithm is formulated by modifying the objective function in the fuzzy C-means algorithm to include a multiplier field, which allows the centroids for each class to vary across the image. First and second order regularization terms ensure that the multiplier field is both slowly varying and smooth. An iterative algorithm that minimizes the objective function is described, and its efficacy is demonstrated on several test images.

An algorithm is presented for the fuzzy segmentation of two-dimensional (2-D) and three-dimensional (3-D) multispectral magnetic resonance (MR) images that have been corrupted by intensity inhomogeneities, also known as shading artifacts. The algorithm is an extension of the 2-D adaptive fuzzy C-means algorithm (2-D AFCM) presented in previous work by the authors. This algorithm models the intensity inhomogeneities as a gain field that causes image intensities to smoothly and slowly vary through the image space. It iteratively adapts to the intensity inhomogeneities and is completely automated. In this paper, we fully generalize 2-D AFCM to three-dimensional (3-D) multispectral images. Because of the potential size of 3-D image data, we also describe a new faster multigrid-based algorithm for its implementation. We show, using simulated MR data, that 3-D AFCM yields lower error rates than both the standard fuzzy C-means (FCM) algorithm and two other competing methods, when segmenting corrupted images. Its efficacy is further demonstrated using real 3-D scalar and multispectral MR brain images.

Reconstructing the geometry of the human cerebral cortex from MR images is an important step in both brain mapping and surgical path planning applications. Difficulties with imaging noise, partial volume averaging, image intensity inhomogeneities, convoluted cortical structures, and the requirement to preserve anatomical topology make the development of accurate automated algorithms particularly challenging. In this paper we address each of these problems and describe a systematic method for obtaining a surface representation of the geometric central layer of the human cerebral cortex. Using fuzzy segmentation, an isosurface algorithm, and a deformable surface model, the method reconstructs the entire cortex with the correct topology, including deep convoluted sulci and gyri. The method is largely automated and its results are robust to imaging noise, partial volume averaging, and image intensity inhomogeneities. The performance of this method is demonstrated, both qualitatively and quantitatively, and the results of its application to six subjects and one simulated MR brain volume are presented.

Magnetic resonance imaging provides the unique capability to produce artificial, high-contrast features called "tags" that are invaluable for tracking tissue motion. Each tag corresponds to an initially planar surface embedded in the tissue that deforms with tissue motion. By tracking tag surface deformation, quantitative analysis of tissue motion can be performed. Here, the authors present a method for tag surface tracking that applies specifically to tag surfaces embedded in the wall of the left ventricle. The method addresses two key issues: first, the full spatial extent of tag surfaces in 3-D space must be inferred from 2-D images, and second, within the images, noise leads to uncertainty in tag positions. The authors address these issues by framing tag surface tracking as an estimation problem given the observed image data. The estimates are obtained using a stochastic model of tag deformation and a recursive algorithm that simultaneously filters over time and smoothly interpolates between images.

This paper introduces a new image processing technique for rapid analysis and visualization of tagged cardiac magnetic resonance (MR) images. The method is based on the use of isolated spectral peaks in SPAMM-tagged magnetic resonance images. The authors call the phase of an image corresponding to just one of these peaks an angle image, and show that except for a phase-wrapping artifact, an angle image is linearly related to a component of the three-dimensional motion. Using one or more angle images, the authors show how to synthesize conventional tag lines, reconstruct displacement fields for small motions, calculate the optical flow between successive temporal images, and calculate two-dimensional strain. The authors demonstrate the performance of this approach on both real and simulated tagged MR images.

Tagged magnetic resonance imaging (MRI) has shown great promise in non-invasive analysis of heart motion. To replace implanted markers as a gold standard, however, tagged MRI must be able to track a sparse set of material points, so-called material markers, with high accuracy. This paper presents a new method for generating accurate motion estimates over a sparse set of material points using standard, parallel-tagged MR images. Each tracked point is located at the intersection of three tag surfaces, each of which is estimated using a thin-plate spline. The intersections are determined by an iterative alternating projections algorithm for which a proof of convergence is provided. The resulting data sets are compatible with applications developed to exploit implanted marker data. One set of these material markers from a normal human volunteer is examined in detail using several methods to visualize the markers. Numerical results that include additional studies are also discussed. Finally, an error analysis is presented using a computer-simulated left ventricle for which material markers are tracked with an RMS error of approximately 0.2 mm for typical imaging parameters and noise levels.

Active contours, or snakes, are used extensively in computer vision and image processing applications, particularly to locate object boundaries. A new type of external force for active contours, called gradient vector flow (GVF) was introduced recently to address problems associated with initialization and poor convergence to boundary concavities. GVF is computed as a diffusion of the gradient vectors of a gray-level or binary edge map derived from the image. In this paper, we generalize the GVF formulation to include two spatially varying weighting functions. This improves active contour convergence to long, thin boundary indentations, while maintaining other desirable properties of GVF, such as an extended capture range. The original GVF is a special case of this new generalized GVF (GGVF) model. An error analysis for active contour results on simulated test images is also presented. Aktive Umrisse, oder Schlangen, werden vielfach in Computervision- und Bildverarbeitungs-Anwendungen benutzt, um insbesondere Objektgrenzen zu lokalisieren. Ein neuer Typ äußerer Kräfte für aktive Umrisse, Gradient Vector Flow (GVF) genannt, wurde kürzlich eingeführt, um Probleme anzusprechen, die mit Initialisierung und schlechter Konvergenz zu Grenzkonkavitäten zusammenhängen. GVF wird als eine Diffusion des Gradientenvektors einer Graustufen- oder `Binary Edge'-Karte berechnet, die aus dem Bild gewonnen werden. In diesem Artikel verallgemeinern wir die GVF Formulierung, so daß zwei räumlich variierende Gewichtsfunktionen engneschlossen werden. Dies verbessert die Konvergenz aktiver Umrisse zu langen, dünnen Grenzmarkierungen, während andere wünschenswerte Eigenschaften des GVF, wie erweiterter Einfangbereich, erhalten bleiben. Das ursprüngliche GVF ist ein Spezialfall dieses neuen verallgemeinerten GVF (GGVF) Modells. Eins Fehleranalyse von Ergebnissen aktiver Umrisse mit simulierten Testbildern wird ebenfalls präsentiert. Les contours actifs, ou serpents (snakes), sont utilisés intensivement en vision par ordinateur et pour les applications de traitement d'images, particulièrement pour localiser les contours d'objects. Un nouveau type de force externe pour les contours actifs, appelè flux de vecteurs gradients (FVG) a été introduit récemment pour traiter les problémes associés à l'initialisation et la faible convergence vers des concavités dans les contours. Le FVG est calculé comme une diffusion des vecteurs gradients d'une carte des contours d'une image en niveaux de gris ou binaire. Dans cet article, nous généralisons la formulation du FVG pour y inclure deux fonctions de poids à variation spatiale. Ceci améliore la convergence des contours actifs vers les indentations de contours fines et longues, tout en maintenant les autres propriétés intéressantes des FVG comme la plage de capture étendue. Les FVG originaux sont un cas particulier des modèles de FVG généralisés. Une analyse de l'erreur des résultats de contours actifs sur des images de test synthétiques est aussi présentée.

Snakes, or active contours, are used extensively in computer vision and image processing applications, particularly to locate object boundaries. Problems associated with initialization and poor convergence to boundary concavities, however, have limited their utility. This paper presents a new external force for active contours, largely solving both problems. This external force, which we call gradient vector flow (GVF), is computed as a diffusion of the gradient vectors of a gray-level or binary edge map derived from the image. It differs fundamentally from traditional snake external forces in that it cannot be written as the negative gradient of a potential function, and the corresponding snake is formulated directly from a force balance condition rather than a variational formulation. Using several two-dimensional (2-D) examples and one three-dimensional (3-D) example, we show that GVF has a large capture range and is able to move snakes into boundary concavities.

Snakes, or active contours, are used extensively in computer vision and image processing applications, particularly to locate object boundaries. Problems associated with initialization and poor convergence to concave boundaries, however, have limited their utility. This paper develops a new external force for active contours, largely solving both problems. This external force, which we call gradient vector flow (GVF) is computed as a diffusion of the gradient vectors of a gray-level or binary edge map derived from the image. The resultant field has a large capture range and forces active contours into concave regions. Examples on simulated images and one real image are presented.

MR tagging has shown great promise for detailed noninvasive cardiac motion imaging. We consider the use of low-frequency tags coupled with gradient-based optical flow estimation to compute cardiac motion. A multiple constraint optical flow method for tagged MRI is formulated by exploiting the Fourier content of the tagged images. The method is validated on simulated tagged data.

This paper considers the problem of vector tomography on an arbitrary bounded domain in three dimensions. Previous work has given the formulas for the reconstructed scalar and vector potential functions in relation to the vector field values inside the domain and on the boundary of the domain. Using these formulas. It is shown that the curl-free component can be reconstructed using only one probe measurement, and the divergence-free component can be reconstructed using only two probe measurements. No boundary measurements are necessary.

A procedure for estimating the joint probability density function (pdf) of T1, T2 and proton spin density (PD) for gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF) in the brain is presented. The pdf's have numerous applications, including the study of tissue parameter variability in pathology and across populations. The procedure requires a multispectral, spin echo magnetic resonance imaging (MRI) data set of the brain. It consists of five automated steps: (i) preprocess the data to remove extracranial tissue using a sequence of image processing operators; (ii) estimate T1, T2 and PD by fitting the preprocessed data to an imaging equation; (iii) perform a fuzzy c-means clustering on the same preprocessed data to obtain a spatial map representing the membership value of the three tissue classes at each pixel location; (iv) reject estimates which are not from pure tissue or have poor fits in the parameter estimation, and classify the remaining estimates as either GM, WM or CSF; (v) compute statistics on the classified estimates to obtain a probability mass function and a Gaussian joint pdf of the tissue parameters for each tissue class. Some preliminary results are shown comparing computed pdf's of young, elderly and Alzheimer's subjects. Two brief examples applying the joint pdf's to pulse sequence optimization and generation of computational phantoms are also provided.

A general active contour formulation is considered and a convexity analysis of its energy function is presented. Conditions under which this formulation has a unique solution are derived; these conditions involve both the active contour energy potential and the regularization parameters. This analysis is then applied to four particular active contour formulations, revealing important characteristics of their convexity, and suggesting that external potentials involving center of mass computations may be better behaved than the usual potentials based on image gradients. Most importantly, our analysis provides an explanation for the poor convergence behavior at concave boundaries and suggests an alternate algorithm for approaching these types of boundaries.

In tagged magnetic resonance imaging of the human heart, different frequency patterns can be imposed and imaged as the heart moves. It is advantageous to use low-frequency patterns to improve both image acquisition time and the performance of gradient-based optical flow motion analysis methods. This paper explores the effects of three factors-image prefiltering, multiresolution optical flow, and multifrequency tag patterns through a serious of controlled simulation experiments. We show that use of image prefiltering in a multiresolution optical flow approach achieves significant performance gains, and permits a factor of four reduction in imaging time, without the need for multifrequency tag patterns.

We consider the classical optical flow algorithm due to Horn and Schunck (1981) for estimating the motion of brightness patterns between image pairs. We use a modified smoothness condition based on the divergence and curl of the velocity field. In previous work, we have developed well-posed stochastic state-space models for these optical flow methods in two dimensions. This paper extends our results to 3-D. We first show that by using the first order div-curl spline, it is not possible to obtain a first order linear differential well-posed model in 3-D. Next, we employ the second order div-curl spline smoothness condition and develop well-posed state-space models.

We consider Suter's (see Proc. CVPR94, Seattle, p.939-948, 1994) DIV-CURL optical flow methods, wherein the problem of computing a velocity field from an image sequence is regularized using smoothness conditions based on the divergence and curl of the field. In particular, we develop stochastic formulations of DIV-CURL splines using the linear smoothing theory of Adams, Willsky, and Levy. Our models are shown to be well posed and thus can be used in both simulating and estimating velocity fields having known stochastic properties. As a special case, our stochastic model reduces to that developed by Rougee, Levy, and Willsky (1984) for the classical Horn and Schunck's (1981) optical flow.

Vector tomography is the reconstruction of vector fields from measurements of their projections. In previous work, it has been shown that the reconstruction of a general three-dimensional (3-D) vector field is possible from the so-called inner product measurements. It has also been shown how the reconstruction of either the irrotational or solenoidal component of a vector field can be accomplished with fewer measurements than that required for the full field. The present paper makes three contributions. First, in analogy to the two-dimensional (2-D) approach of Norton (1988), several 3-D projection theorems are developed. These lead directly to new vector field reconstruction formulas that are convolution backprojection formulas. It is shown how the local reconstruction property of these 3-D reconstruction formulas permits reconstruction of point flow or of regional flow from a limited data set. Second, simulations demonstrating 3-D reconstructions, both local and nonlocal, are presented. Using the formulas derived herein and those derived in previous work, these results demonstrate the reconstruction of the irrotational and solenoidal components, their potential functions, and the field itself from simulated inner product measurement data. Finally, it is shown how 3-D inner product measurements can be acquired using a magnetic resonance scanner.

A new two-stage approach for nonlinear brain image registration is proposed. In the first stage, an active contour algorithm is used to establish a homothetic one-to-one map between a set of region boundaries in two images to be registered. This mapping is used in the second step: a two-dimensional transformation which is based on an elastic body deformation. This method is tested by registering magnetic resonance images to atlas images.

The routine use of a single radionuclide for patient imaging in nuclear medicine can be complemented by studies employing two tracers to examine two different processes in a single organ, most frequently by simultaneous imaging of both radionuclides in two different energy windows. In addition, simultaneous transmission/emission imaging with dual-radionuclides has been described, with one radionuclide used for the transmission study and a second for the emission study. There is thus currently considerable interest in dual-radionuclide imaging. A major problem with all dual-radionuclide imaging is the "crosstalk" between the two radionuclides. Such crosstalk frequently occurs, because scattered radiation from the higher energy radionuclide is detected in the lower energy window, and because the lower energy radionuclide may have higher energy emissions which are detected in the higher energy window. The authors have previously described the use of Fourier-based restoration filtering in single photon emission computed tomography (SPECT) and positron emission tomography (PET) to improve quantitative accuracy by designing a Wiener or other Fourier filter to partially restore the loss of contrast due to scatter and finite spatial resolution effects. The authors describe here the derivation and initial validation of an extension of such filtering for dual-radionuclide imaging that simultaneously 1) improves contrast in each radionuclide's "direct" image, 2) reduces image noise, and 3) reduces the crosstalk contribution from the other radionuclide. This filter is based on a vector version of the Wiener filter, which is shown to be superior [in the minimum mean square error (MMSE) sense] to the sequential application of separate crosstalk and restoration filters.

In this paper, we examine optical flow methods applicable under variable brightness conditions. In particular, we develop a stochastic formulation of Gennert and Negahdaripour's (1987) optical flow algorithm. Our model is shown to be well-posed and thus can be used in both simulating and estimating velocity fields with certain known stochastic properties. This model is used to simulate flow fields of different physical properties, and Gennert and Negahdaripour's method is demonstrated to work well for these synthesized fields under variable brightness conditions.

A frequency domain performance analysis of Horn and Schunck's (1981) optical flow (HSOF) algorithm for estimation of deformable motion is presented. Noise sources in the algorithm are modeled using the discrete Fourier transform of the brightness pattern. This noise model along with the estimation error covariance function derived in previous work is used to derive an expression for the expected performance of the optical flow estimate that is valid for an arbitrary discrete brightness pattern. Simulation results are presented that demonstrate the validity of our methods and show that HSOF is more accurate that the optical flow estimate of Anandan for certain low-frequency patterns.

A new active contour model for finding and mapping the outer cortex in brain images is developed. A cross-section of the brain cortex is modeled as a ribbon, and a constant speed mapping of its spine is sought. A variational formulation, an associated force balance condition, and a numerical approach are proposed to achieve this goal. The primary difference between this formulation and that of snakes is in the specification of the external force acting on the active contour. A study of the uniqueness and fidelity of solutions is made through convexity and frequency domain analyses, and a criterion for selection of the regularization coefficient is developed. Examples demonstrating the performance of this method on simulated and real data are provided.

Magnetic resonance (MR) tagging has shown great potential for noninvasive measurement of the motion of a beating heart. In MR tagged images, the heart appears with a spatially encoded pattern that moves with the tissue. The position of the tag pattern in each frame of the image sequence can be used to obtain a measurement of the 3-D displacement field of the myocardium. The measurements are sparse, however, and interpolation is required to reconstruct a dense displacement field from which measures of local contractile performance such as strain can be computed. Here, the authors propose a method for estimating a dense displacement field from sparse displacement measurements. Their approach is based on a multidimensional stochastic model for the smoothness and divergence of the displacement field and the Fisher estimation framework. The main feature of this method is that both the displacement field model and the resulting estimate equation are defined only on the irregular domain of the myocardium. The authors' methods are validated on both simulated and in vivo heart data.

Magnetic resonance (MR) tagging is a technique used to impose a known intensity pattern, called a tag pattern, on an object by spatially modulating the object's proton spin density before the image is acquired. Since the tag pattern moves with the object, MR tagging is particularly useful in the estimation of deformable motion from MR image sequences because some tag patterns can resolve motion ambiguities such as the aperture problem and improve the accuracy of optical flow motion estimates. We present an experimental validation of the tag pattern optimization method of Denney and Prince (see Proc. of the IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, p.V225-V228, Minneapolis, April 1993, and IEEE Transactions on Image Processing, vol.3, no.2, p.178-191, March 1994) in a practical MR imaging environment. This experiment includes a study of the effects of temporal sampling on the optimal frequency that was not done by Denney et. al. Our experimental results show that the tag pattern optimization method of Denney et. al. is valid in a practical MR imaging application.>

Three-dimensional (3-D) tomographic measurements of vector flow fields are possible using acoustical, optical, and magnetic resonance imaging. A general mathematical framework for this type of data acquisition called the probe transform has been studied by the author in previous work. It has been shown that the irrotational field component can be reconstructed using only one probe direction, and that the solenoidal field component can be reconstructed using only two probe directions. Previously, the reconstruction formulas were of the backprojection-convolution type; that is, first backproject then convolve in 3-D. In this paper a convolution-backprojection formula, which has significant computational savings, is developed. Numerical simulations are also given to demonstrate the performance of this new approach.>

Estimation accuracy of Horn and Schunck's (1981) classical optical flow algorithm depends on many factors including the brightness pattern of the measured images. Since some applications can select brightness functions with which to "paint" the object, it is desirable to know what patterns will lead to the best motion estimates. The paper presents a method for determining this pattern a priori using mild assumptions about the velocity field and imaging process. The method is based on formulating Horn and Schunck's algorithm as a linear smoother and rigorously deriving an expression for the corresponding error covariance function. The authors then specify a scalar performance measure and develop an approach to select an optimal brightness function which minimizes this performance measure from within a parametrized class. Conditions for existence of an optimal brightness function are also given. The resulting optimal performance is demonstrated using simulations, and a discussion of these results and potential future research is given.

Inner product probe measurements are defined for tomographic reconstruction of 3-D vector fields. It is shown that one set of measurements is required to reconstruct an irrotational field, two are required to reconstruct a solenoidal field and special probes are required to reconstruct the components of an arbitrary field.

Tracking magnetic resonance tags in myocardial tissue promises to be an effective tool for the assessment of myocardial motion. The authors describe a hierarchy of image processing steps which rapidly detects both the contours of the myocardial boundaries of the left ventricle and the tags within the myocardium. The method works on both short axis and long axis images containing radial and parallel tag patterns, respectively. Left ventricular boundaries are detected by first removing the tags using morphological closing and then selecting candidate edge points. The best inner and outer boundaries are found using a dynamic program that minimizes a nonlinear combination of several local cost functions. Tags are tracked by matching a template of their expected profile using a least squares estimate. Since blood pooling, contiguous and adjacent tissue, and motion artifacts sometimes cause detection errors, a graphical user interface was developed to allow user correction of anomalous points. The authors present results on several tagged images of a human. A fully automated run generally finds the endocardial boundary and the tag lines extremely well, requiring very little manual correction. The epicardial boundary sometimes requires more intervention to obtain an acceptable result. These methods are currently being used in the analysis of cardiac strain and as a basis for the analysis of alternate tag geometries.

Two new adaptive active contour algorithms for the extraction and mapping of the skeleton of a thick curve are described. They are based on conditions which guarantee uniqueness and fidelity of the solution. Both algorithms modify the regularization constant K/sub o/ in an attempt to maintain convexity of the energy function while simultaneously improving the fidelity of the result. The first algorithm changes K/sub o/ over time while the second adapts K/sub o/ spatially. Both algorithms are evaluated on experiments with synthetic curves; both demonstrate an improved performance compared to a fixed-parameter active contour algorithm.>

The authors describe and demonstrate a hierarchical reconstruction algorithm for use in noisy and limited-angle or sparse-angle tomography. The algorithm estimates an object's mass, center of mass, and convex hull from the available projections, and uses this information, along with fundamental mathematical constraints, to estimate a full set of smoothed projections. The mass and center of mass estimates are made using a least squares estimator derived from the principles of consistency of the Radon transform. The convex hull estimate is produced by first estimating the positions of support lines of the object from each available projection and then estimating the overall convex hull using prior shape information. Estimating the position of two support lines from a single projection is accomplished using a generalized likelihood ratio technique for estimating jumps in linear systems. Results for simulated objects in a variety of measurement situations are shown, and several possible extensions to this work are discussed.

A method that simultaneously identifies the central layer of the human cortex and maps it onto the interval

In the typical optical flow problem, the brightness function is an inherent part of the underlying physical process. In the case of magnetic resonance imaging of the heart, however, it is possible to modulate the brightness function using tagging techniques. Experimental evidence suggests that when a sinusoidal brightness function is used, there is an optimum frequency. The reasons for an optimal frequency in one dimension are investigated by formulating the optical flow problem as an optimal linear smoothing problem. A method based on minimizing the theoretical smoothed error variance is proposed for finding the optimal frequency a priori. Simulations are presented for a Wiener process velocity that show the existence of an optimal frequency and the ability of the proposed method to determine this frequency a priori.>

A method for reconstructing motion from sequences of tagged magnetic resonance (MR) images is presented. MR tagging is used to create a spatial pattern of varying magnetization so that objects which may otherwise have constant intensity are textured, which reduces the motion ambiguity associated with the aperture problem in computer vision. To compensate for the decay of the tag pattern, a new optical flow algorithm is developed and implemented. The resulting estimated velocity field is then used to recursively update the implied motion reference map over time, thereby tracking the motion of individual particles. If a segmentation of the object is known at the time the tag pattern is created, then an object may be selectively tracked, using the estimated reference map to update the object's position as time progresses. Results are shown for both simulated and actual MR phantom data.

A method to reconstruct motion from sequences of tagged magnetic resonance (MR) images is presented. MR tagging is used to create a spatial pattern of varying magnetization so that objects which may otherwise have constant intensity are textured; this reduces the motion ambiguity associated with the aperture problem in computer vision. To account for the decay of the tag pattern, a new optical flow algorithm is developed and implemented. Velocity fields estimated using this algorithm are used to recursively update the implied motion reference map over time, thereby tracking the motion of individual particles. If a segmentation of the object is known at the time the tag pattern is created, then an object may be selectively tracked, using the estimated reference map to update the object's position as time progresses. Results are shown for actual MR phantom data.>

In this paper we present several algorithms for reconstructing 2D convex sets given support line measurements for which the angles are known precisely but the lateral displacements are noisy. We extend the algorithms given in a previous paper by explicitly incorporating prior information about the shape of the objects to be reconstructed. We develop the Scale-Invariant algorithms, which incorporate prior shape information by defining prior probabilities on support vectors, where a support vector is a vector formed from the lateral displacements of a particular set of support lines of an object. We also develop the Ellipse-Based algorithms, which either assume or jointly estimate the parameters of an ellipse, given prior distributions that favor ellipses. In order to relate the support vector prior probability to the expected shape of an object we develop a vector decomposition called the Size/Shape/Shift decomposition, which helps to provide insight into the detailed geometric relationship between support vectors and 2D convex objects. We then use the maximum a posteriori criterion to determine the specific form of the support vector estimator. The computations involve a quadratic programming optimization stage, which is used to determine one component of the decomposition, and either a line search or a conjugate gradient stage, which is used to determine the remaining components. The performance of the algorithms is demonstrated using simulated support line measurements of an ellipse.

Algorithms are proposed for reconstructing convex sets given noisy support line measurements. It is observed that a set of measured support lines may not be consistent with any set in the plane. A theory of consistent support lines which serves as a basis for reconstruction algorithms that take the form of constrained optimization algorithms is developed. The formal statement of the problem and constraints reveals a rich geometry that makes it possible to include prior information about object position and boundary smoothness. The algorithms, which use explicit noise models and prior knowledge, are based on maximum-likelihood and maximum a posteriori estimation principles and are implemented using efficient linear and quadratic programming codes. Experimental results are presented. This research sets the stage for a more general approach to the incorporation of prior information concerning the estimation of object shape.>

An overview of the architecture and implementation of a model-based vision system developed for object recognition with SAR (synthetic aperture radar) data is presented. The initial implementation of the system is currently being used to develop experimental results to guide refinements and enhancements. The system uses detailed analytic prediction models and capable description algorithms, with all knowledge and uncertainty consistently represented. The prediction component automatically generates integrated hierarchical representations of both structural and appearance information, and represents an important step toward automatic object recognition. The recognition system architecture features modular computational agents that support distributed, localized control, with the ability to extract and use object-specific knowledge from the prediction database. The system will serve as a testbed for model-based vision research to allow experimentation with new algorithms and alternative recognition approaches.>

The authors describe and demonstrate a hierarchical reconstruction algorithm for use in noisy and limited-angle or sparse-angle tomography. The algorithm estimates the object's mass, center of mass, and convex hull from the available projections, and uses this information, along with fundamental mathematical constraints, to estimate a full set of smoothed projections. The mass and center of mass are estimated using a maximum-likelihood (ML) estimator derived from the principles of consistency of the Radon transform. The convex hull estimate is produced by first estimating the positions of support lines of the object from each available projection and then estimating the overall convex hull using ML or maximum a posteriori (MAP) techniques. The position of two support lines from a single projection is estimated using either a generalized likelihood ratio technique for estimating jumps in linear systems or a support-width penalty method that uses Akaike's model-order estimation technique.>

Presents a method to reconstruct images from finite sets of noisy projections which are available only over limited or sparse angles. The method solves a constrained optimization problem to find a maximum a posteriori (MAP) estimate of the full 2-D Radon transform of the object, using prior knowledge of object mass, center of mass, and convex support, and information about fundamental constraints and smoothness of the Radon transform. This efficient primal-dual algorithm consists of an iterative local relaxation stage which solves a partial differential equation in Radon-space, followed by a simple Lagrange multiplier update stage. The object is reconstructed using convolution backprojection applied to the Radon transform estimate.>