Auto-segmentation quality or accuracy influences their clinical usefulness. However, currently widely utilized segmentation metrics (e.g., Dice Coefficient (DC) and Hausdorff Distance (HD)) cannot effectively express the manual mending effort required when utilizing auto-segmentation results in clinical practice. In this article, we explore ways of evaluating auto-segmentations with clinical efficiency considerations in mind. The time required for correcting auto-segmentations by experts is recorded to indicate ground-truth mending effort. Extended from our previous work, five explicitly-defined metrics are studied in detail for their ability to predict mending effort. More importantly, we explore the use of deep learning networks to provide an implicit metric, which predict mending effort using auto-segmentation masks and original images as input. A 3-institution evaluation is conducted with 7 different anatomic organs in the setting of auto-contouring for radiation therapy planning. Among the five explicit metrics, one form of the proposed Mendability Index (MIhd) shows the best performance to indicate the mending effort for sparse objects with 6.2-14.4% error, while one form of HD (sHD) performs best when assessing large non-sparse objects. Interestingly, while the explicit metrics all require ground truth segmentations for estimating mending effort, the implicit models obtained via deep learning are effective in predicting mending efforts (with 2.9-12.9% error) without the need for ground-truth segmentations and directly from the given image plus the auto-segmentations. We conclude that once effort-predicting deep models are created, it is feasible to assess the clinical usability of new segmentation models, going beyond bench technical evaluation commonly done via explicit metrics.

Eye tracking research is important in computer vision because it can help us understand how humans interact with the visual world. Specifically for high-risk applications, such as in medical imaging, eye tracking can help us to comprehend how radiologists and other medical professionals search, analyze, and interpret images for diagnostic and clinical purposes. Hence, the application of eye tracking techniques in disease classification has become increasingly popular in recent years. Contemporary works usually transform gaze information collected by eye tracking devices into visual attention maps (VAMs) to supervise the learning process. However, this is a time-consuming preprocessing step, which stops us from applying eye tracking to radiologists’ daily work. To solve this problem, we propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into VAMs. In GazeGNN, to directly integrate eye gaze into image classification, we create a unified representation graph that models both images and gaze pattern information. With this benefit, we develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature. This achievement demonstrates the practicality and feasibility of integrating real-time eye tracking techniques into the daily work of radiologists. To our best knowledge, GazeGNN is the first work that adopts GNN to integrate image and eye-gaze data. Our experiments on the public chest X-ray dataset show that our proposed method exhibits the best classification performance compared to existing methods. The code is available at https://github.com/ukaukaaaa/GazeGNN.

Evaluating medical imaging segmentation is a very complex problem. Several papers proposed methodologies and different metrics pursuing more reliable and unbiased procedures. In this paper, we propose a novel accuracy metric which is more balanced than the well known Dice and Jaccard coefficients. We also prove mathematically that the proposed metric generalizes Dice, Jaccard and the previously proposed Balanced Dice and Balanced Jaccard coefficients. Our experiments show that significant changes in brain tissue segmentation evaluation results are noticed as we applied our new metric.

Many papers are published every year containing new methodologies for brain tissue segmentation in magnetic resonance images. The evaluation of these methods is fundamental to understand their behavior and to observe their weak and strong points. Even though improvements have been proposed, the analysis of the segmentation results can still lead to incorrect conclusions. This paper contains an investigation of the state-of-the-art in brain tissue segmentation evaluation, which includes tissue classification or segmentation, handling partial volume effect, and evaluation metrics. It uncovers previously unnoticed pitfalls and proposes standard procedures to avoid them. Experiments show that the proposed evaluation strategy gives a better insight about the method's strong and weak points.

In the above paper (ibid., vol. 33, no. 12, pp. 2293-2310, Dec. 2014), Awais Mansoor was incorrectly indicated as the corresponding author. Ulas Bagci should have been indicated as the corresponding author.

In this study, we propose a novel pathological lung segmentation method that takes into account neighbor prior constraints and a novel pathology recognition system. Our proposed framework has two stages; during stage one, we adapted the fuzzy connectedness (FC) image segmentation algorithm to perform initial lung parenchyma extraction. In parallel, we estimate the lung volume using rib-cage information without explicitly delineating lungs. This rudimentary, but intelligent lung volume estimation system allows comparison of volume differences between rib cage and FC based lung volume measurements. Significant volume difference indicates the presence of pathology, which invokes the second stage of the proposed framework for the refinement of segmented lung. In stage two, texture-based features are utilized to detect abnormal imaging patterns (consolidations, ground glass, interstitial thickening, tree-inbud, honeycombing, nodules, and micro-nodules) that might have been missed during the first stage of the algorithm. This refinement stage is further completed by a novel neighboring anatomy-guided segmentation approach to include abnormalities with weak textures, and pleura regions. We evaluated the accuracy and efficiency of the proposed method on more than 400 CT scans with the presence of a wide spectrum of abnormalities. To our best of knowledge, this is the first study to evaluate all abnormal imaging patterns in a single segmentation framework. The quantitative results show that our pathological lung segmentation method improves on current standards because of its high sensitivity and specificity and may have considerable potential to enhance the performance of routine clinical tasks.

We introduce an image segmentation algorithm GCsummax, which combines, in a novel manner, the strengths of two popular algorithms: Relative Fuzzy Connectedness (RFC) and (standard) Graph Cut (GC). We show, both theoretically and experimentally, that GCsummaxpreserves robustness of RFC with respect to the seed choice (thus, avoiding “shrinking problem” of GC), while keeping GC's bigger control over “leaking though the weak boundary.” The theoretical analysis of GCsummaxis greatly facilitated by our recent theoretical results that RFC belongs to the Generalized GC (GGC) segmentation algorithms framework. In our implementation of GCsummaxwe use, as a subroutine, a version of RFC algorithm (based on Image Foresting Transform) that runs (provably) in linear time with respect to the image size. This results in GCsummaxrunning in a time close to linear.

Cerebral parenchyma segmentation in newborn magnetic resonance (MR) images is crucial for developing computer-aided diagnosis systems in newborn cerebral diseases. However, there is limited number of studies on newborn brain MR image analysis. This study presents a novel method for fully automatically segmenting the cerebral parenchyma region using scale-based fuzzy connected image segmentation and fuzzy object models. The proposed method evaluates object affinity and homogeneity using the MR signal, and employs a fuzzy object model, which is built from training datasets. We have evaluated the proposed method based on 10 newborn MR images with subject revised age between -1 month and 2 months. These studies indicate that the use of a fuzzy object model is effective in improving the segmentation accuracy.

In this paper, we propose a novel method based on a strategic combination of the active appearance model (AAM), live wire (LW), and graph cuts (GCs) for abdominal 3-D organ segmentation. The proposed method consists of three main parts: model building, object recognition, and delineation. In the model building part, we construct the AAM and train the LW cost function and GC parameters. In the recognition part, a novel algorithm is proposed for improving the conventional AAM matching method, which effectively combines the AAM and LW methods, resulting in the oriented AAM (OAAM). A multiobject strategy is utilized to help in object initialization. We employ a pseudo-3-D initialization strategy and segment the organs slice by slice via a multiobject OAAM method. For the object delineation part, a 3-D shape-constrained GC method is proposed. The object shape generated from the initialization step is integrated into the GC cost computation, and an iterative GC-OAAM method is used for object delineation. The proposed method was tested in segmenting the liver, kidneys, and spleen on a clinical CT data set and also on the MICCAI 2007 Grand Challenge liver data set. The results show the following: 1) The overall segmentation accuracy of true positive volume fraction TPVF > 94.3% and false positive volume fraction can be achieved; 2) the initialization performance can be improved by combining the AAM and LW; 3) the multiobject strategy greatly facilitates initialization; 4) compared with the traditional 3-D AAM method, the pseudo-3-D OAAM method achieves comparable performance while running 12 times faster; and 5) the performance of the proposed method is comparable to state-of-the-art liver segmentation algorithm. The executable version of the 3-D shape-constrained GC method with a user interface can be downloaded from http://xinjianchen.wordpress.com/research/.

Segmentation of anatomical structures from medical images is a challenging problem, which depends on the accurate recognition (localization) of anatomical structures prior to delineation. This study generalizes anatomy segmentation problem via attacking two major challenges: 1) automatically locating anatomical structures without doing search or optimization, and 2) automatically delineating the anatomical structures based on the located model assembly. For 1), we propose intensity weighted ball-scale object extraction concept to build a hierarchical transfer function from image space to object (shape) space such that anatomical structures in 3-D medical images can be recognized without the need to perform search or optimization. For 2), we integrate the graph-cut (GC) segmentation algorithm with prior shape model. This integrated segmentation framework is evaluated on clinical 3-D images consisting of a set of 20 abdominal CT scans. In addition, we use a set of 11 foot MR images to test the generalizability of our method to the different imaging modalities as well as robustness and accuracy of the proposed methodology. Since MR image intensities do not possess a tissue specific numeric meaning, we also explore the effects of intensity nonstandardness on anatomical object recognition. Experimental results indicate that: 1) effective recognition can make the delineation more accurate; 2) incorporating a large number of anatomical structures via a model assembly in the shape model improves the recognition and delineation accuracy dramatically; 3) ball-scale yields useful information about the relationship between the objects and the image; 4) intensity variation among scenes in an ensemble degrades object recognition performance.

Active shape models (ASM) are widely employed for recognizing anatomic structures and for delineating them in medical images. In this paper, a novel strategy called oriented active shape models (OASM) is presented in an attempt to overcome the following five limitations of ASM: 1) lower delineation accuracy, 2) the requirement of a large number of landmarks, 3) sensitivity to search range, 4) sensitivity to initialization, and 5) inability to fully exploit the specific information present in the given image to be segmented. OASM effectively combines the rich statistical shape information embodied in ASM with the boundary orientedness property and the globally optimal delineation capability of the live wire methodology of boundary segmentation. The latter characteristics allow live wire to effectively separate an object boundary from other nonobject boundaries with similar properties especially when they come very close in the image domain. The approach leads to a two-level dynamic programming method, wherein the first level corresponds to boundary recognition and the second level corresponds to boundary delineation, and to an effective automatic initialization method. The method outputs a globally optimal boundary that agrees with the shape model if the recognition step is successful in bringing the model close to the boundary in the image. Extensive evaluation experiments have been conducted by utilizing 40 image (magnetic resonance and computed tomography) data sets in each of five different application areas for segmenting breast, liver, bones of the foot, and cervical vertebrae of the spine. Comparisons are made between OASM and ASM based on precision, accuracy, and efficiency of segmentation. Accuracy is assessed using both region-based false positive and false negative measures and boundary-based distance measures. The results indicate the following: 1) The accuracy of segmentation via OASM is considerably better than that of ASM; 2) The number of landmarks can be reduced by a factor of 3 in OASM over that in ASM; 3) OASM becomes largely independent of search range and initialization becomes automatic. All three benefits of OASM ensue mainly from the severe constraints brought in by the boundary-orientedness property of live wire and the globally optimal solution found by the 2-level dynamic programming algorithm.

Image intensity standardization is a postprocessing method designed for correcting acquisition-to-acquisition signal intensity variations (nonstandardness) inherent in magnetic resonance (MR) images. Inhomogeneity correction is a process used to suppress the low frequency background nonuniformities (inhomogeneities) of the image domain that exist in MR images. Both these procedures have important implications in MR image analysis. The effects of these postprocessing operations on improvement of image quality in isolation has been well documented. However, the combined effects of these two processes on MR images and how the processes influence each other have not been studied thus far. In this paper, we evaluate the effect of inhomogeneity correction followed by standardization and vice-versa on MR images in order to determine the best sequence to follow for enhancing image quality. We conducted experiments on several clinical and phantom data sets (nearly 4000 three-dimensional MR images were analyzed) corresponding to four different MRI protocols. Different levels of artificial nonstandardness, and different models and levels of artificial background inhomogeneity were used in these experiments. Our results indicate that improved standardization can be achieved by preceding it with inhomogeneity correction. There is no statistically significant difference in image quality obtained between the results of standardization followed by correction and that of correction followed by standardization from the perspective of inhomogeneity correction. The correction operation is found to bias the effect of standardization. We demonstrate this bias both qualitatively and quantitatively by using two different methods of inhomogeneity correction. We also show that this bias in standardization is independent of the specific inhomogeneity correction method used. The effect of this bias due to correction was also seen in magnetization transfer ratio (MTR) images, which are naturally endowed with the standardness property. Standardization, on the other hand, does not seem to influence the correction operation. It is also found that longer sequences of repeated correction and standardization operations do not considerably improve image quality. These results were found to hold for the clinical and the phantom data sets, for different MRI protocols, for different levels of artificial nonstandardness, for different models and levels of artificial inhomogeneity, for different correction methods, and for images that were endowed with inherent standardness as well as for those that were standardized by using the intensity standardization method. Overall, we conclude that inhomogeneity correction followed by intensity standardization is the best sequence to follow from the perspective of both image quality and computational efficiency.

In tomographic imagery, partial volume effects (PVEs) cause several artifacts in volume renditions. In X-ray computed tomography (CT), for example, soft-tissue-like pseudo structures appear in bone-to-air and bone-to-fat interfaces. Further, skin, which is identical to soft tissue in terms of CT number, obscures the rendition of the latter. The purpose of this paper is to demonstrate these phenomena and to provide effective solutions that yield significantly improved renditions. We introduce two methods that detect and classify voxels with PVE in X-ray CT. Further, a method is described to automatically peel skin so that PVE-resolved renditions of bone and soft tissue reveal considerably more detail. In the first method to address PVE, called the fraction measure (FM) method, the fraction of each tissue material in each voxel v is estimated by taking into account the intensities of the voxels neighboring v. The second method, called uncertainty principle (UP) method, is based on the following postulate (Saha and Udupa, 2001): In any acquired image, voxels with the highest uncertainty occur in the vicinity of object boundaries. The removal of skin is achieved by means of mathematical morphology. Volume renditions have been created before and after applying the methods for several patient CT datasets. A mathematical phantom experiment involving different levels of PVE has been conducted by adding different degrees of noise and blurring. A quantitative evaluation is done utilizing the mathematical phantom and clinical CT data wherein an operator carefully masked out voxels with PVE in the segmented images. All results have demonstrated the enhanced quality of display of bone and soft tissue after applying the proposed methods. The quantitative evaluations indicate that more than 98% of the voxels with PVE are removed by the two methods and the second method performs slightly better than the first. Further, skin peeling vividly reveals fine details in the soft tissue structures.

In many medical imaging applications, due to the limited field of view of imaging devices, acquired images often include only a part of a structure. In such situations, it is impossible to guarantee that the images will contain exactly the same physical extent of the structure at different scans, which leads to difficulties in registration and in many other tasks, such as the analysis of the morphology, architecture, and kinematics of the structures. To facilitate such analysis, we developed a general method, referred to as iso-shaping, that generates structures of the same shape from segmented image sequences. The basis for this method is to automatically find a set of key points, called shape centers, in the segmented partial anatomic structure such that these points are present in all images and that they represent the same physical location in the object, and then trim the structure using these points as reference. The application area considered here is the analysis of the morphology, architecture, and kinematics of the joints of the foot from magnetic resonance images acquired at different joint positions and load conditions. The accuracy of the method is analyzed by utilizing ten data sets for iso-shaping the tibia and the fibula via four evaluative experiments. The analysis indicates that iso-shaping produces results as predicted by the theoretical framework.

Provides an overview of the technical articles and features presented in this issue.

Image segmentation-the process of defining objects in images-remains the most challenging problem in image processing despite decades of research. Many general methodologies have been proposed to date to tackle this problem. An emerging framework that has shown considerable promise recently is that of fuzzy connectedness. Images are by nature fuzzy. Object regions manifest themselves in images with a heterogeneity of image intensities owing to the inherent object material heterogeneity, and artifacts such as blurring, noise and background variation introduced by the imaging device. In spite of this gradation of intensities, knowledgeable observers can perceive object regions as a gestalt. The fuzzy connectedness framework aims at capturing this notion via a fuzzy topological notion called fuzzy connectedness which defines how the image elements hang together spatially in spite of their gradation of intensities. In defining objects in a given image, the strength of connectedness between every pair of image elements is considered, which in turn is determined by considering all possible connecting paths between the pair. In spite of a high combinatorial complexity, theoretical advances in fuzzy connectedness have made it possible to delineate objects via dynamic programming at close to interactive speeds on modern PCs. This paper gives a tutorial review of the fuzzy connectedness framework delineating the various advances that have been made. These are illustrated with several medical applications in the areas of Multiple Sclerosis of the brain, magnetic resonance (MR) and computer tomographic (CT) angiography, brain tumor, mammography, upper airway disorders in children, and colonography.

Finite Normal Mixture (FNM) model-based image segmentation techniques adopt the following detection-estimation-classification paradigm: 1) detect the number of image regions by using theoretical information criteria; 2) estimate model parameters by using expectation-maximization (EM)/classification-maximization (CM) algorithms; and 3) classify pixels into regions by using various classifiers. This paper presents a theoretical framework to evaluate the performance of this class of image segmentation techniques. For the detection performance, probabilities of over-detection and under-detection of the number of image regions are defined, and the associated formulae in terms of model parameters and image quality are derived. For the estimation performance, both EM and CM algorithms are showed to produce asymptotically unbiased ML estimates of model parameters in the case of no-overlap. Cramer-Rao bounds of variances of these estimates are derived. For the classification performance, misclassification probability for the Bayesian classifier is defined, and a simple formula based on parameter estimates and classified data is derived to evaluate segmentation errors. This evaluation method provides both theoretically approachable accuracy limits of the techniques and practically achievable performance of the given images. Theoretical and experimental results are in good agreement and indicate that, for images of moderate quality, the detection operation is robust, the parameter estimates are accurate, and the segmentation errors are small.

We present a new class of approaches for rigid-body registration and their evaluation in studying multiple sclerosis (MS) via multiprotocol magnetic resonance imaging (MRI). Three pairs of rigid-body registration algorithms were implemented, using cross-correlation and mutual information (MI), operating on original gray-level images, and utilizing the intermediate images resulting from our new scale-based method. In the scale image, every voxel has the local "scale" value assigned to it, defined as the radius of the largest ball centered at the voxel with homogeneous intensities. Three-dimensional image data of the head were acquired from ten MS patients for each of six MRI protocols. Images in some of the protocols were acquired in registration. The registered pairs were used as ground truth. Accuracy and consistency of the six registration methods were measured within and between protocols for known amounts of misregistrations. Our analysis indicates that there is no "best" method. For medium misregistration, the method using MI, for small add large misregistration the method using normalized cross-correlation performs best. For high-resolution data the correlation method and for low-resolution data the MI method, both using the original gray-level images, are the most consistent. We have previously demonstrated the use of local scale information in fuzzy connectedness segmentation and image filtering. Scale may also have potential for image registration as suggested by this work.

The notion of fuzzy connectedness captures the idea of "hanging-togetherness" of image elements in an object by assigning a strength of connectedness to every possible path between every possible pair of image elements. This concept leads to powerful image segmentation algorithms based on dynamic programming whose effectiveness has been demonstrated on 1,000s of images in a variety of applications. In the previous framework, a fuzzy connected object is defined with a threshold on the strength of connectedness. We introduce the notion of relative connectedness that overcomes the need for a threshold and that leads to more effective segmentations. The central idea is that an object gets defined in an image because of the presence of other co-objects. Each object is initialized by a seed element. An image element c is considered to belong to that object with respect to whose reference image element c has the highest strength of connectedness. In this fashion, objects compete among each other utilizing fuzzy connectedness to grab membership of image elements. We present a theoretical and algorithmic framework for defining objects via relative connectedness and demonstrate utilizing the theory that the objects defined are independent of reference elements chosen as long as they are not in the fuzzy boundary between objects. An iterative strategy is also introduced wherein the strongest relative connected core parts are first defined and iteratively relaxed to conservatively capture the more fuzzy parts subsequently. Examples from medical imaging are presented to illustrate visually the effectiveness of relative fuzzy connectedness. A quantitative mathematical phantom study involving 160 images is conducted to demonstrate objectively the effectiveness of relative fuzzy connectedness.DisclaimerA claim of priority in research and publication appeared on page 1486 in the paper "Relative Fuzzy Connectedness and Object Definition: Theory, Algorithms, and Applications in Image Segmentation" by J.K. Udupa, P.K. Saha, and' R.A. Lotufo (IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 11, pp. 1485-1500,Nov. 2002) with respect to the paper "Multiseeded Segmentation Using Fuzzy Connectedness by G.T. Herman and B.M. Carvalho" (IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 5, pp. 460-474, May 2001). Furthermore, the wording of this claim suggests professional misconduct on the part of G.T. Herman and B.M. Carvalho. Responsibility for the content of published papers rests with the authors. The peer review process is intended to determine the overall significance of the technical contribution of a manuscript. The peer review process does not provide a way to validate every statement in a manuscript. In particular, the IEEE has not validated the claim referred to in the first paragraph above. The IEEE regrets publishing this unauthenticated statement and the pain that such publication may have caused to G.T. Herman and B.M. Carvalho.

mm , and for the clear layer, and mm , mm , and for highly scattering layers. The dotted line in the figure shows the backward scattering pulse profile of the homogeneous slab without the clear layer at the 20 mm away from laser illumination points. The profile changes by adding the clear layer especially in the trailing edge of the pulse retaining the peak amplitude. When absorption coefficient in the third layer was increased from mm to mm , the decay of the reflectance is also increased as shown in the figure. This means that the absorption changes inside the clear layer are estimated by the time profile changes of the reflectance. The sensitivity for the absorption changes becomes maximum when source detector separation is 20 mm. The mean path lengths are 200, 660, 825, and 870 ps for the source detector spacing 10, 20, 30, and 37.5 mm, respectively. The mean path length saturate for the spacing larger than 25 mm, which are coincident with results reported by Monte Carlo and FEM simulation [9]. V. C ONCLUSION FDTD method has been successfully formulated for analysis of pulse responses in biological tissues. Boundary conditions have been defined using fluence rate at the scattering and no scattering material interfaces. The conditions to give stabilities for numerical solutions have been become clear in terms of scattering coefficients and mean cosine of scattering angles. Using the formulation, the reflectance of three-layered slabs containing a clear layer has been calculated. As a result, it has been become clear that absorption coefficient changes of the highly scattering media beyond the clear layer are estimated from the time profiles of the reflectance. The analysis is easily extended to calculate pulse responses in tissues with 3-D inhomogeneous optical parameters, which is effective for image reconstruction in practical diffused optical tomography.

In this paper, we identify and describe the key ideas, concepts and software related to Medical Imaging that emerged from our research group, which is commonly known by the acronym MIPG, denoting Medical Image Processing Group. For the purpose of our presentation, we consider the entire discipline of Medical Imaging to consist of the areas of acquisition, postprocessing, evaluation, and application. We are indeed fortunate to be able to discuss seminal research from our group in each of these areas. In Section II, we identify relevant moments of inception and describe our past contributions. Our achievements, current activities, and future directions are summarized in Section III, Section IV, and Section V, respectively; the keys to our success are listed in Section V.

Thresholding is a popular image segmentation method that converts a gray-level image into a binary image. The selection of optimum thresholds has remained a challenge over decades. Besides being a segmentation tool on its own, often it is also a step in many advanced image segmentation techniques in spaces other than the image space. We introduce a thresholding method that accounts for both intensity-based class uncertainty-a histogram-based property-and region homogeneity-an image morphology-based property. A scale-based formulation is used for region homogeneity computation. At any threshold, intensity-based class uncertainty is computed by fitting a Gaussian to the intensity distribution of each of the two regions segmented at that threshold. The theory of the optimum thresholding method is based on the postulate that objects manifest themselves with fuzzy boundaries in any digital image acquired by an imaging device. The main idea here is to select that threshold at which pixels with high class uncertainty accumulate mostly around object boundaries. To achieve this, a threshold energy criterion is formulated using class-uncertainty and region homogeneity such that, at any image location, a high energy is created when both class uncertainty and region homogeneity are high or both are low. Finally, the method selects that threshold which corresponds to the minimum overall energy. The method has been compared to a maximum segmented image information method. Superiority of the proposed method was observed both qualitatively on clinical medical images as well as quantitatively on 250 realistic phantom images generated by adding different degrees of blurring, noise, and background variation to real objects segmented from clinical images.

This paper presents a near-automatic process for separating vessels from background and other clutter as well as for separating arteries and veins in contrast-enhanced magnetic resonance angiographic (CE-MRA) image data, and an optimal method for three-dimensional visualization of vascular structures. The separation process utilizes fuzzy connected object delineation principles and algorithms. The first step of this separation process is the segmentation of the entire vessel structure from the background and other clutter via absolute fuzzy connectedness. The second step is to separate artery from vein within this entire vessel structure via iterative relative fuzzy connectedness. After seed voxels are specified inside artery and vein in the CE-MRA image, the small regions of the bigger aspects of artery and vein are separated in the initial iterations, and further detailed aspects of artery and vein are included in later iterations. At each iteration, the artery and vein compete among themselves to grab membership of each voxel in the vessel structure based on the relative strength of connectedness of the voxel in the artery and vein. This approach has been implemented in a software package for routine use in a clinical setting and tested on 133 CE-MRA studies of the pelvic region and two studies of the carotid system from six different hospitals. In all studies, unified parameter settings produced correct artery-vein separation. When compared with manual segmentation/separation, our algorithms were able to separate higher order branches, and therefore produced vastly more details in the segmented vascular structure. The total operator and computer time taken per study is on the average about 4.5 min. To date, this technique seems to be the only image processing approach that can be routinely applied for artery and vein separation.

Image acquisition techniques often suffer from low signal-to-noise ratio (SNR) and/or contrast-to-noise ratio (CNR). Although many acquisition techniques are available to minimize these, post acquisition filtering is a major off-line image processing technique commonly used to improve the SNR and CNR. A major drawback of filtering is that it often diffuses/blurs important structures along with noise. In this paper, we introduce two scale-based filtering methods that use local structure size or "object scale" information to arrest smoothing around fine structures and across even low-gradient boundaries. The first of these methods uses a weighted average over a scale-dependent neighborhood while the other employs scale-dependent diffusion conductance to perform filtering. Both methods adaptively modify the degree of filtering at any image location depending on local object scale. Object scale allows us to accurately use a restricted homogeneity parameter for filtering in regions with fine details and in the vicinity of boundaries while a generous parameter in the interiors of homogeneous regions. Qualitative experiments based on both phantoms and patient magnetic resonance images show significant improvements using the scale-based methods over the extant anisotropic diffusive filtering method in preserving fine details and sharpness of object boundaries. Quantitative analyses utilizing 25 phantom images generated under a range of conditions of blurring, noise, and background variation confirm the superiority of the new scale-based approaches.

Studies reported in the literature indicate that breast cancer risk is associated with mammographic densities. An objective, repeatable, and a quantitative measure of risk derived from mammographic densities will be of considerable use in recommending alternative screening paradigms and/or preventive measures. However, image processing efforts toward this goal seem to be sparse in the literature, and automatic and efficient methods do not seem to exist. In this paper, we describe and validate an automatic and reproducible method to segment dense tissue regions from fat within breasts from digitized mammograms using scale-based fuzzy connectivity methods. Different measures for characterizing mammographic density are computed from the segmented regions and their robustness in terms of their linear correlation across two different projections--cranio-caudal and medio-lateral-oblique--are studied. The accuracy of the method is studied by computing the area of mismatch of segmented dense regions using the proposed method and using manual outlining. A comparison between the mammographic density parameter taking into account the original intensities and that just considering the segmented area indicates that the former may have some advantages over the latter.

This paper extends a previously reported theory and algorithms for object definition based on fuzzy connectedness. In this approach, a strength of connectedness is determined between every pair of image elements. This is done by considering all possible connecting paths between the two elements in each pair. The strength assigned to a particular path is defined as the weakest affinity between successive pairs of elements along the path. Affinity specifies the degree to which elements hang together locally in the image. Although the theory allowed any neighborhood size for affinity definition, it did not indicate how this was to be selected. By bringing object scale into the framework in this paper, not only the size of the neighborhood is specified but also it is allowed to change in different parts of the image. This paper argues that scale-based affinity, and hence connectedness, is natural in object definition and demonstrates that this leads to more effective object segmentation. The approach presented here considers affinity to consist of two components. The homogeneity-based component indicates the degree of affinity between image elements based on the homogeneity of their intensity properties. The object-feature-based component captures the degree of closeness of their intensity properties to some expected values of those properties for the object. A family of non-scale-based and scale-based affinity relations are constructed dictated by how we envisage the two components to characterize objects. A simple and effective method for giving a rough estimate of scale at different locations in the image is presented. The original theoretical and algorithmic framework remains more-or-less the same but considerably improved segmentations result. The method has been tested in several applications qualitatively. A quantitative statistical comparison between the non-scale-based and the scale-based methods was made based on 250 phantom images. These were generated from 10 patient MR brain studies by first segmenting the objects, then setting up appropriate intensity levels for the object and the background, and then by adding five different levels for each of noise and blurring and a fixed slow varying background component. Both the statistical and the subjective tests clearly indicate that the scale-based method is superior to the non-scale-based method in capturing details and in robustness to noise. It is also shown, based on these phantom images, that any (global) optimum threshold selection method will perform inferior to the fuzzy connectedness methods described in this paper.

We have been developing user-steered image segmentation methods for situations which require considerable human assistance in object definition. In the past, we have presented two paradigms, referred to as live-wire and live-lane, for segmenting 2D/3D/4D object boundaries in a slice-by-slice fashion, and demonstrated that live-wire and live-lane are more repeatable, with a statistical significance level of P < 0.03, and are 1.5-2.5 times faster, with a statistical significance level of P < 0.02, than manual tracing. In this paper, we introduce a 3D generalization of the live-wire approach for segmenting 3D/4D object boundaries which further reduces the time spent by the user in segmentation. In a 2D live-wire, given a slice, for two specified points (pixel vertices) on the boundary of the object, the best boundary segment is the minimum-cost path between the two points, described as a set of oriented pixel edges. This segment is found via Dijkstra's algorithm as the user anchors the first point and moves the cursor to indicate the second point. A complete 2D boundary is identified as a set of consecutive boundary segments forming a "closed", "connected", "oriented" contour. The strategy of the 3D extension is that, first, users specify contours via live-wiring on a few slices that are orthogonal to the natural slices of the original scene. If these slices are selected strategically, then we have a sufficient number of points on the 3D boundary of the object to subsequently trace optimum boundary segments automatically in all natural slices of the 3D scene. A 3D object boundary may define multiple 2D boundaries per slice. The points on each 2D boundary form an ordered set such that when the best boundary segment is computed between each pair of consecutive points, a closed, connected, oriented boundary results. The ordered set of points on each 2D boundary is found from the way the users select the orthogonal slices. Based on several validation studies involving segmentation of the bones of the foot in MR images, we found that the 3D extension of live-wire is more repeatable, with a statistical significance level of P < 0.0001, and 2-6 times faster, with a statistical significance level of P < 0.01, than the 2D live-wire method, and 3-15 times faster than manual tracing.

We have been developing general user steered image segmentation strategies for routine use in applications involving a large number of data sets. In the past, we have presented three segmentation paradigms: live wire, live lane, and a three-dimensional (3-D) extension of the live-wire method. In this paper, we introduce an ultra-fast live-wire method, referred to as live wire on the fly, for further reducing user's time compared to the basic live-wire method. In live wire, 3-D/four-dimensional (4-D) object boundaries are segmented in a slice-by-slice fashion. To segment a two-dimensional (2-D) boundary, the user initially picks a point on the boundary and all possible minimum-cost paths from this point to all other points in the image are computed via Dijkstra's algorithm. Subsequently, a live wire is displayed in real time from the initial point to any subsequent position taken by the cursor. If the cursor is close to the desired boundary, the live wire snaps on to the boundary. The cursor is then deposited and a new live-wire segment is found next. The entire 2-D boundary is specified via a set of live-wire segments in this fashion. A drawback of this method is that the speed of optimal path computation depends on image size. On modestly powered computers, for images of even modest size, some sluggishness appears in user interaction, which reduces the overall segmentation efficiency. In this work, we solve this problem by exploiting some known properties of graphs to avoid unnecessary minimum-cost path computation during segmentation. In live wire on the fly, when the user selects a point on the boundary the live-wire segment is computed and displayed in real time from the selected point to any subsequent position of the cursor in the image, even for large images and even on low-powered computers. Based on 492 tracing experiments from an actual medical application, we demonstrate that live wire on the fly is 1.3-31 times faster than live wire for actual segmentation for varying image sizes, although the pure computational part alone is found to be about 120 times faster.

One of the major drawbacks of magnetic resonance imaging (MRI) has been the lack of a standard and quantifiable interpretation of image intensities. Unlike in other modalities, such as X-ray computerized tomography, MR images taken for the same patient on the same scanner at different times may appear different from each other due to a variety of scanner-dependent variations and, therefore, the absolute intensity values do not have a fixed meaning. We have devised a two-step method wherein all images (independent of patients and the specific brand of the MR scanner used) can be transformed in such a way that for the same protocol and body region, in the transformed images similar intensities will have similar tissue meaning. Standardized images can be displayed with fixed windows without the need of per-case adjustment. More importantly, extraction of quantitative information about healthy organs or about abnormalities can be considerably simplified. This paper introduces and compares new variants of this standardizing method that can help to overcome some of the problems with the original method.

The purpose of this work is to compare the speed of isosurface rendering in software with that using dedicated hardware. Input data consist of 10 different objects from various parts of the body and various modalities (CT, MR, and MRA) with a variety of surface sizes (up to 1 million voxels/2 million triangles) and shapes. The software rendering technique consists of a particular method of voxel-based surface rendering, called shell rendering. The hardware method is OpenGL-based and uses the surfaces constructed from our implementation of the Marching Cubes algorithm. The hardware environment consists of a variety of platforms, including a Sun Ultra I with a Creator3D graphics card and a Silicon Graphics Reality Engine II, both with polygon rendering hardware, and a 300 MHz Pentium PC. The results indicate that the software method (shell rendering) was 18 to 31 times faster than any hardware rendering methods. This work demonstrates that a software implementation of a particular rendering algorithm (shell rendering) can outperform dedicated hardware. We conclude that, for medical surface visualization, expensive dedicated hardware engines are not required. More importantly, available software algorithms (shell rendering) on a 300 MHz Pentium PC outperform the speed of rendering via hardware engines by a factor of 18 to 31.

Image interpolation is an important operation that is widely used in medical imaging, image processing, and computer graphics. A variety of interpolation methods are available in the literature. However, their systematic evaluation is lacking. In a previous paper, we presented a framework for the task-independent comparison of interpolation methods based on certain image-derived figures of merit using a variety of medical image data pertaining to different parts of the human body taken from different modalities. In this work, we present an objective task-specific framework for evaluating interpolation techniques. The task considered is how the interpolation methods influence the accuracy of quantification of the total volume of lesions in the brain of multiple sclerosis (MS) patients. Sixty lesion-detection experiments coming from ten patient studies, two subsampling techniques and the original data, and three interpolation methods are carried out, along with a statistical analysis of the results.

The purpose of this work is to study the architecture of the rearfoot using in vivo MR image data. Each data set used in this study is made of sixty sagittal slices of the foot acquired in a 1.5-T commercial GE MR system. We use the live-wire method to delineate boundaries and form the surfaces of the bones. In the first part of this work, we describe a new method to characterize the three-dimensional (3-D) relationships of four bones of the peritalar complex and apply this description technique to data sets from ten normal subjects and from seven pathological cases. In the second part, we propose a procedure to classify feet, based on the values of these new architectural parameters. We conclude that this noninvasive method offers a unique tool to characterize the 3-D architecture of the feet in live patients, based on a set of new architectural parameters. This can be integrated into a set of tools to improve diagnosis and treatment of foot malformations.

In multidimensional image analysis, there are, and will continue to be, situations wherein automatic image segmentation methods fail, calling for considerable user assistance in the process. The main goals of segmentation research for such situations ought to be (i) to provideeffective controlto the user on the segmentation processwhileit is being executed, and (ii) to minimize the total user's time required in the process. With these goals in mind, we present in this paper two paradigms, referred to aslive wireandlive lane, for practical image segmentation in large applications. For both approaches, we think of the pixel vertices and oriented edges as forming a graph, assign a set of features to each oriented edge to characterize its ``boundariness,'' and transform feature values to costs. We provide training facilities and automatic optimal feature and transform selection methods so that these assignments can be made with consistent effectiveness in any application. In live wire, the user first selects an initial point on the boundary. For any subsequent point indicated by the cursor, an optimal path from the initial point to the current point is found and displayed in real time. The user thus has a live wire on hand which is moved by moving the cursor. If the cursor goes close to the boundary, the live wire snaps onto the boundary. At this point, if the live wire describes the boundary appropriately, the user deposits the cursor which now becomes the new starting point and the process continues. A few points (live-wire segments) are usually adequate to segment the whole 2D boundary. In live lane, the user selects only the initial point. Subsequent points are selected automatically as the cursor is moved within a lane surrounding the boundary whose width changes as a function of the speed and acceleration of cursor motion. Live-wire segments are generated and displayed in real time between successive points. The users get the feeling that the curve snaps onto the boundary as and while they roughly mark in the vicinity of the boundary. We describe formal evaluation studies to compare the utility of the new methods with that of manual tracing based on speed and repeatability of tracing and on data taken from a large ongoing application. The studies indicate that the new methods are statistically significantly more repeatable and 1.5–2.5 times faster than manual tracing.

To aid in the display, manipulation, and analysis of biomedical image data, they usually need to he converted to data of isotropic discretization through the process of interpolation. Traditional techniques consist of direct interpolation of the grey values. When user interaction is called for in image segmentation, as a consequence of these interpolation methods, the user needs to segment a much greater (typically 4-10x) amount of data. To mitigate this problem, a method called shape-based interpolation of binary data was developed 121. Besides significantly reducing user time, this method has been shown to provide more accurate results than grey-level interpolation. We proposed an approach for the interpolation of grey data of arbitrary dimensionality that generalized the shape-based method from binary to grey data. This method has characteristics similar to those of the binary shape-based method. In particular, we showed preliminary evidence that it produced more accurate results than conventional grey-level interpolation methods. In this paper, concentrating on the three-dimensional (3-D) interpolation problem, we compare statistically the accuracy of eight different methods: nearest-neighbor, linear grey-level, grey-level cubic spline, grey-level modified cubic spline, Goshtasby et al., and three methods from the grey-level shape-based class. A population of patient magnetic resonance and computed tomography images, corresponding to different parts of the human anatomy, coming from different three-dimensional imaging applications, are utilized for comparison. Each slice in these data sets is estimated by each interpolation method and compared to the original slice at the same location using three measures: mean-squared difference, number of sites of disagreement, and largest difference. The methods are statistically compared pairwise based on these measures. The shape-based methods statistically significantly outperformed all other methods in all measures in all applications considered here with a statistical relevance ranging from 10% to 32% (mean = 15%) for mean-squared difference.

Multiple sclerosis (MS) is a disease of the white matter. Magnetic resonance imaging (MRI) is proven to be a sensitive method of monitoring the progression of this disease and of its changes due to treatment protocols. Quantification of the severity of the disease through estimation of MS lesion volume via MR imaging is vital for understanding and monitoring the disease and its treatment. This paper presents a novel methodology and a system that can be routinely used for segmenting and estimating the volume of MS lesions via dual-echo fast spin-echo MR imagery. A recently developed concept of fuzzy objects forms the basis of this methodology. An operator indicates a few points in the images by pointing to the white matter, the grey matter, and the cerebro-spinal fluid (CSF). Each of these objects is then detected as a fuzzy connected set. The holes in the union of these objects correspond to potential lesion sites which are utilized to detect each potential lesion as a three-dimensional (3-D) fuzzy connected object. These objects are presented to the operator who indicates acceptance/rejection through the click of a mouse button. The number and volume of accepted lesions is then computed and output. Based on several evaluation studies, we conclude that the methodology is highly reliable and consistent, with a coefficient of variation (due to subjective operator actions) of 0.9% (based on 20 patient studies, three operators, and two trials) for volume and a mean false-negative volume fraction of 1.3%, with a 95% confidence interval of 0%-2.8% (based on ten patient studies).

Images are by nature fuzzy. Approaches to object information extraction from images should attempt to use this fact and retain fuzziness as realistically as possible. In past image segmentation research, the notion of “hanging togetherness” of image elements specified by their fuzzy connectedness has been lacking. We present a theory of fuzzy objects forn-dimensional digital spaces based on a notion of fuzzy connectedness of image elements. Although our definitions lead to problems of enormous combinatorial complexity, the theoretical results allow us to reduce this dramatically, leading us to practical algorithms for fuzzy object extraction. We present algorithms for extracting a specified fuzzy object and for identifying all fuzzy objects present in the image data. We demonstrate the utility of the theory and algorithms in image segmentation based on several practical examples all drawn from medical imaging.

Shape-based interpolation as applied to binary images causes the interpolation process to be influenced by the shape of the object. It accomplishes this by first applying a distance transform to the data. This results in the creation of a grey-level data set in which the value at each point represents the minimum distance from that point to the surface of the object. (By convention, points inside the object are assigned positive values; points outside are assigned negative values.) This distance transformed data set is then interpolated using linear or higher-order interpolation and is then thresholded at a distance value of zero to produce the interpolated binary data set. Here, the authors describe a new method that extends shape-based interpolation to grey-level input data sets. This generalization consists of first lifting the n-dimensional (n-D) image data to represent it as a surface, or equivalently as a binary image, in an (n+1)-dimensional [(n+1)-D] space. The binary shape-based method is then applied to this image to create an (n+1)-D binary interpolated image. Finally, this image is collapsed (inverse of lifting) to create the n-D interpolated grey-level data set. The authors have conducted several evaluation studies involving patient computed tomography (CT) and magnetic resonance (MR) data as well as mathematical phantoms. They all indicate that the new method produces more accurate results than commonly used grey-level linear interpolation methods, although at the cost of increased computation.

In many imaging applications, boundaries of objects need to be identified in multidimensional digital image data for the visualization and analysis of object information captured in the images. This article addresses the question of how to define boundaries in multidimensional digital spaces so that they are "closed" and connected, and so that they partition the digital space into an interior set that is connected and an exterior set that is connected. Using adjacency relations defined on the elements of the digital space and on boundary elements, we prove some basic results relating to these properties of boundaries. We examine in detail some specific boundary element adjacency relations and present efficient algorithms that track boundaries defined in binary images of any (finite) diinensionality. We conclude with two conjectures relating to the connectedness of boundaries.

There are many imaging modalities (e.g., medical imaging scanners) that capture information about internal structures and generate three-dimensional (3D) digital images of the distribution of some physical property of the material of the structure. Such images have been found to be very useful in analyzing the form and function of the structure and in detecting and correcting deformities in the structure. Visualization of 3D structures is an essential component of such analyses. One commonly used approach to visualization consists of identifying the structure of interest, forming its surfaces, and then rendering the surfaces on a two-dimensional screen. This paper addresses the surface formation problem assuming that object identification has already been done and a 3D binary image is available that represents the structure. For the existing 3D boundary tracking algorithms, the user has to somehow specify each surface that is to be tracked. Often, the 3D image consists of many surfaces of interest. Their manual specification is very tedious and may be impossible if the structure is of complex shape. This paper describes a methodology for automatically tracking all boundary surfaces—i.e., labelling boundary surfaces—in the given 3D image. The algorithms also generate additional information from which the 3D connected components in the image are trivially obtained. Examples from medical imaging are included to illustrate the usefulness of the new methodology.

The development of image processing algorithms for time-varying imagery and computerized tomography data calls for generalization of the concepts of adjacency, connectivity, boundary, etc., to three and four-dimensional discrete spaces. This paper defines these basic concepts in unified terminology and presents algorithms for a boundary detection task in multidimensional space. The performance of these algorithms is discussed with respect to theoretical maximum complexity, and is illustrated with simulated computerized tomography data.

A new approach to machine representation and analysis of three-dimensional objects is presented. The representation, based on the notion of "skeleton" of an object leads to a scheme for comparing two given object views for shape relations. The objects are composed of long, thin, rectangular prisms joined at their ends. The input picture to the program is the digitized line drawing portraying the three-dimensional object. To compare two object views, two characteristic vertices called "cardinal point" and "end-cardinal point," occurring consistently at the bends and open ends of the object are detected. The skeletons are then obtained as a connected path passing through these points. The shape relationships between the objects are then obtained from the matching characteristics of their skeletons. The method explores the possibility of a more detailed and finer analysis leading to detection of features like symmetry, asymmetry and other shape properties of an object.

Computer simulation of one aspect of the human visual perceptual capabilities, viz., the ability to visualize object views that result from known rotations of familiar objects is considered. The objects considered are composed of long, thin, and narrow rectangular prisms connected at their ends. From the digitized binary picture of the line drawing portraying a given three-dimensional object a description is generated in terms of a set of "cardinal points" (which are key points occurring at the bends and open ends of the object) and a set of direction vectors associated with them. The skeleton for the given view is obtained as an ordered sequence of "slope-code" numbers by tracking between the cardinal points. A set of transformations is defined on the sequence and the direction vectors to get the skeleton of the view that results from rotation. From a knowledge of the transformed direction vectors the machine builds up faces around the bends and ends of the skeleton to complete the new view. As far as machine visualization is concerned, it is shown that object rotations of only integer multiples of π/2 need be considered.