Deep learning (DL) methods have shown remarkable successes in medical image segmentation, often using large amounts of annotated data for model training. However, acquiring a large number of diverse labeled 3D medical image datasets is highly difficult and expensive. Recently, mask propagation DL methods were developed to reduce the annotation burden on 3D medical images. For example, Sli2Vol [59] proposed a self-supervised framework (SSF) to learn correspondences by matching neighboring slices via slice reconstruction in the training stage; the learned correspondences were then used to propagate a labeled slice to other slices in the test stage. But, these methods are still prone to error accumulation due to the inter-slice propagation of reconstruction errors. Also, they do not handle discontinuities well, which can occur between consecutive slices in 3D images, as they emphasize exploiting object continuity. To address these challenges, in this work, we propose a new SSF, called Sli2Vol+, for segmenting any anatomical structures in 3D medical images using only a single annotated slice per training and testing volume. Specifically, in the training stage, we first propagate an annotated 2D slice of a training volume to the other slices, generating pseudo-labels (PLs). Then, we develop a novel Object Estimation Guided Correspondence Flow Network to learn reliable correspondences between consecutive slices and corresponding PLs in a self-supervised manner. In the test stage, such correspondences are utilized to propagate a single annotated slice to the other slices of a test volume. We demonstrate the effectiveness of our method on various medical image segmentation tasks with different datasets, showing better generalizability across different organs, modalities, and modals. Code is available at https://github.com/adlsn/Sli2Volplus

The Vision Transformer has attained remarkable success in various computer vision applications. However, the large computational costs and complex design limit its ability in handling large feature maps. Existing research predominantly focuses on constraining attention to small local regions, which reduces the number of tokens attending the attention computation while overlooking computational demands caused by the feed-forward layer in the Vision Transformer block. In this paper, we introduce Group Vision Transformer (GVT), a relatively simple and efficient variant of Vision Transformer, aiming to improve attention computation. The core idea of our model is to divide and group the entire Transformer layer, instead of only the attention part, into multiple independent branches. This approach offers two advantages: (1) It helps reduce parameters and computational complexity; (2) it enhances the diversity of the learned features. We conduct comprehensive analysis of the impact of different numbers of groups on model performance, as well as their influence on parameters and computational complexity. Our proposed GVT demonstrates competitive performances in several common vision tasks. For example, our GVT-Tiny model achieves 84.8% top-1 accuracy on ImageNet-1K, 51.4% box mAP and 45.2% mask mAP on MS COCO object detection and instance segmentation, and 50.1% mIoU on ADE20K semantic segmentation, outperforming the CAFormer-S36 model by 0.3% in ImageNet-1K top-1 accuracy, 1.2% in box mAP, 1.0% in mask mAP on MS COCO object detection and instance segmentation, and 1.2% in mIoU on ADE20K semantic segmentation, with similar model parameters and computational complexity. Code is accessible at https://github.com/yaoppeng/GVT.

Modern deep neural networks have achieved great successes in medical image analysis. However, the features captured by convolutional neural networks (CNNs) or Transformers tend to be optimized for pixel intensities and neglect key anatomical structures such as connected components and loops. In this paper, we propose a persistent homology guided approach (PHG-Net) that explores topological features of objects for medical image classification. For an input image, we first compute its cubical persistence diagram and extract topological features into a vector representation using a small neural network (called the PH module). The extracted topological features are then incorporated into the feature map generated by CNN or Transformer for feature fusion. The PH module is lightweight and capable of integrating topological features into any CNN or Transformer architectures in an end-to-end fashion. We evaluate our PHG-Net on three public datasets and demonstrate its considerable improvements on the target classification tasks over state-of-the-art methods.

This paper considers a novel application of deep AUC maximization (DAM) for multi-instance learning (MIL), in which a single class label is assigned to a bag of instances (e.g., multiple 2D slices of a CT scan for a patient). We address a neglected yet non-negligible computational challenge of MIL in the context of DAM, i.e., bag size is too large to be loaded into GPU memory for backpropagation, which is required by the standard pooling methods of MIL. To tackle this challenge, we propose variance-reduced stochastic pooling methods in the spirit of stochastic optimization by formulating the loss function over the pooled prediction as a multi-level compositional function. By synthesizing techniques from stochastic compositional optimization and non-convex min-max optimization, we propose a unified and provable muli-instance DAM (MIDAM) algorithm with stochastic smoothed-max pooling or stochastic attention-based pooling, which only samples a few instances for each bag to compute a stochastic gradient estimator and to update the model parameter. We establish a similar convergence rate of the proposed MIDAM algorithm as the state-of-the-art DAM algorithms. Our extensive experiments on conventional MIL datasets and medical datasets demonstrate the superiority of our MIDAM algorithm. The method is open-sourced at https://libauc.org/.

Quantitative analysis of neurite growth and morphology is essential for understanding the determinants of neural development and regeneration, however, it is complicated by the labor-intensive process of measuring diverse parameters of neurite outgrowth. Consequently, automated approaches have been developed to study neurite morphology in a high-throughput and comprehensive manner. These approaches include computer-automated algorithms known as 'convolutional neural networks' (CNNs)-powerful models capable of learning complex tasks without the biases of hand-crafted models. Nevertheless, their complexity often relegates them to functioning as 'black boxes.' Therefore, research in the field of explainable AI is imperative to comprehend the relationship between CNN image analysis output and predefined morphological parameters of neurite growth in order to assess the applicability of these machine learning approaches. In this study, drawing inspiration from the field of automated feature selection, we investigate the correlation between quantified metrics of neurite morphology and the image analysis results from NeuriteNet-a CNN developed to analyze neurite growth. NeuriteNet accurately distinguishes images of neurite growth based on different treatment groups within two separate experimental systems. These systems differentiate between neurons cultured on different substrate conditions and neurons subjected to drug treatment inhibiting neurite outgrowth. By examining the model's function and patterns of activation underlying its classification decisions, we discover that NeuriteNet focuses on aspects of neuron morphology that represent quantifiable metrics distinguishing these groups. Additionally, it incorporates factors that are not encompassed by neuron morphology tracing analyses. NeuriteNet presents a novel tool ideally suited for screening morphological differences in heterogeneous neuron groups while also providing impetus for targeted follow-up studies.

Deep AUC Maximization (DAM) is a new paradigm for learning a deep neural network by maximizing the AUC score of the model on a dataset. Most previous works of AUC maximization focus on the perspective of optimization by designing efficient stochastic algorithms, and studies on generalization performance of large-scale DAM on difficult tasks are missing. In this work, we aim to make DAM more practical for interesting real-world applications (e.g., medical image classification). First, we propose a new margin-based min-max surrogate loss function for the AUC score (named as the AUC min-max-margin loss or simply AUC margin loss for short). It is more robust than the commonly used AUC square loss, while enjoying the same advantage in terms of large-scale stochastic optimization. Second, we conduct extensive empirical studies of our DAM method on four difficult medical image classification tasks, namely (i) classification of chest x-ray images for identifying many threatening diseases, (ii) classification of images of skin lesions for identifying melanoma, (iii) classification of mammogram for breast cancer screening, and (iv) classification of microscopic images for identifying tumor tissue. Our studies demonstrate that the proposed DAM method improves the performance of optimizing cross-entropy loss by a large margin, and also achieves better performance than optimizing the existing AUC square loss on these medical image classification tasks. Specifically, our DAM method has achieved the 1st place on Stanford CheXpert competition on Aug. 31, 2020. To the best of our knowledge, this is the first work that makes DAM succeed on large-scale medical image datasets. We also conduct extensive ablation studies to demonstrate the advantages of the new AUC margin loss over the AUC square loss on benchmark datasets. The proposed method is implemented in our open-sourced library LibAUC (www.libauc.org) whose github address is https://github.com/Optimization-AI/LibAUC.

A fully automated knee magnetic resonance imaging (MRI) segmentation method to study osteoarthritis (OA) was developed using a novel hierarchical set of random forests (RF) classifiers to learn the appearance of cartilage regions and their boundaries. A neighborhood approximation forest is used first to provide contextual feature to the second-level RF classifier that also considers local features and produces location-specific costs for the layered optimal graph image segmentation of multiple objects and surfaces (LOGISMOS) framework. Double-echo steady state MRIs used in this paper originated from the OA Initiative study. Trained on 34 MRIs with varying degrees of OA, the performance of the learning-based method tested on 108 MRIs showed significant reduction in segmentation errors ( ) compared with the conventional gradient-based and single-stage RF-learned costs. The 3-D LOGISMOS was extended to longitudinal-3-D (4-D) to simultaneously segment multiple follow-up visits of the same patient. As such, data from all time-points of the temporal sequence contribute information to a single optimal solution that utilizes both spatial 3-D and temporal contexts. 4-D LOGISMOS validation on 108 MRIs from baseline, and 12 month follow-up scans of 54 patients showed significant reduction in segmentation errors ( ) compared with 3-D. Finally, the potential of 4-D LOGISMOS was further explored on the same 54 patients using five annual follow-up scans demonstrating a significant improvement of measuring cartilage thickness ( ) compared with the sequential 3-D approach.

Features of high-risk coronary artery plaques prone to major adverse cardiac events (MACE) were identified by intravascular ultrasound (IVUS) virtual histology (VH). These plaque features are: thin-cap fibroatheroma (TCFA), plaque burden PB ≥ 70%, or minimal luminal area MLA ≤ 4 mm2. Identification of arterial locations likely to later develop such high-risk plaques may help prevent MACE. We report a machine learning method for prediction of future high-risk coronary plaque locations and types in patients under statin therapy. Sixty-one patients with stable angina on statin therapy underwent baseline and one-year follow-up VH-IVUS non-culprit vessel examinations followed by quantitative image analysis. For each segmented and registered VH-IVUS frame pair (${n} =6341$ ), location-specific ($\approx 0.5$ mm) vascular features and demographic information at baseline were identified. Seven independent support vector machine classifiers with seven different feature subsets were trained to predict high-risk plaque types one year later. A leave-one-patient-out cross-validation was used to evaluate the prediction power of different feature subsets. The experimental results showed that our machine learning method predicted future TCFA with correctness of 85.9%, 81.7%, and 77.0% (G-mean) for baseline plaque phenotypes of TCFA, thick-cap fibroatheroma, and non-fibroatheroma, respectively. For predicting PB ≥ 70%, correctness was 80.8% for baseline PB ≥ 70% and 85.6% for 50% ≤ PB2was 81.6% for baseline MLA ≤ 4 mm2and 80.2% for 4 mm22. Location-specific prediction of future high-risk coronary artery plaques is feasible through machine learning using focal vascular features and demographic variables. Our approach outperforms previously reported results and shows the importance of local factors on high-risk coronary artery plaque development.

We propose an effective optimization algorithm for a general hierarchical segmentation model with geometric interactions between segments. Any given tree can specify a partial order over object labels defining a hierarchy. It is well-established that segment interactions, such as inclusion/exclusion and margin constraints, make the model significantly more discriminant. However, existing optimization methods do not allow full use of such models. Generic a-expansion results in weak local minima, while common binary multi-layered formulations lead to non-submodularity, complex high-order potentials, or polar domain unwrapping and shape biases. In practice, applying these methods to arbitrary trees does not work except for simple cases. Our main contribution is an optimization method for the Hierarchically-structured Interacting Segments (HINTS) model with arbitrary trees. Our Path-Moves algorithm is based on multi-label MRF formulation and can be seen as a combination of well-known a-expansion and Ishikawa techniques. We show state-of-the-art biomedical segmentation for many diverse examples of complex trees.

Star-convexity prior is popular for interactive single object segmentation due to its simplicity and amenability to binary graph cut optimization. We propose a more general multi-object segmentation approach. Moreover, each object can be constrained by a more descriptive shape prior, "hedgehog". Each hedgehog shape has its surface normals locally constrained by an arbitrary given vector field, e.g. gradient of the user-scribble distance transform. In contrast to star-convexity, the tightness of our normal constraint can be changed giving better control over allowed shapes. For example, looser constraints, i.e. wider cones of allowed normals, give more relaxed hedgehog shapes. On the other hand, the tightest constraint enforces skeleton consistency with the scribbles. In general, hedgehog shapes are more descriptive than a star, which is only a special case corresponding to a radial vector field and weakest tightness. Our approach has significantly more applications than standard single star-convex segmentation, e.g. in medical data we can separate multiple non-star organs with similar appearances and weak edges. Optimization is done by our modified -expansion moves shown to be submodular for multi-hedgehog shapes.

Automated retinal layer segmentation of optical coherence tomography (OCT) images has been successful for normal eyes but becomes challenging for eyes with retinal diseases if the retinal morphology experiences critical changes. We propose a method to automatically segment the retinal layers in 3-D OCT data with serous retinal pigment epithelial detachments (PED), which is a prominent feature of many chorioretinal disease processes. The proposed framework consists of the following steps: fast denoising and B-scan alignment, multi-resolution graph search based surface detection, PED region detection and surface correction above the PED region. The proposed technique was evaluated on a dataset with OCT images from 20 subjects diagnosed with PED. The experimental results showed the following. 1) The overall mean unsigned border positioning error for layer segmentation is 7.87±3.36 μm , and is comparable to the mean inter-observer variability ( 7.81±2.56 μm). 2) The true positive volume fraction (TPVF), false positive volume fraction (FPVF) and positive predicative value (PPV) for PED volume segmentation are 87.1%, 0.37%, and 81.2%, respectively. 3) The average running time is 220 s for OCT data of 512 × 64 × 480 voxels.

Automated three-dimensional retinal fluid (named symptomatic exudate-associated derangements, SEAD) segmentation in 3D OCT volumes is of high interest in the improved management of neovascular Age Related Macular Degeneration (AMD). SEAD segmentation plays an important role in the treatment of neovascular AMD, but accurate segmentation is challenging because of the large diversity of SEAD size, location, and shape. Here a novel voxel classification based approach using a layer-dependent stratified sampling strategy was developed to address the class imbalance problem in SEAD detection. The method was validated on a set of 30 longitudinal 3D OCT scans from 10 patients who underwent anti-VEGF treatment. Two retinal specialists manually delineated all intraretinal and subretinal fluid. Leave-one-patient-out evaluation resulted in a true positive rate and true negative rate of 96% and 0.16% respectively. This method showed promise for image guided therapy of neovascular AMD treatment.

A novel method is reported for simultaneous registration of location (axial direction) and orientation (circumferential direction) of two intravascular ultrasound (IVUS) pullbacks of the same vessel taken at different times. Monitoring plaque progression or regression (e.g., during lipid treatment) is of high clinical relevance. Our method uses a 3D graph optimization approach, in which the cost function jointly reflects similarity of plaque morphology and plaque/perivascular image appearance. Graph arcs incorporate prior information about temporal correspondence of the two IVUS sequences and limited angular twisting between consecutive IVUS images. Additionally, our approach automatically identifies starting and ending frame pairs in the two IVUS pullbacks. Validation of our method was performed in 29 pairs of IVUS baseline/follow-up pullback sequences consisting of 8 622 IVUS image frames in total. In comparison to manual registration by three experts, the average location and orientation registration errors ranged from 0.72 mm to 0.79 mm and from 7.3(°) to 9.3(°), respectively, all close to the inter-observer variability with no difference being statistically significant (p = NS). Rotation angles determined by our automated approach and expert observers showed high correlation (r(2) of 0.97 to 0.98) and agreed closely (mutual bias between the automated method and expert observers ranged from -1.57(°) to 0.15(°)). Compared with state-of-the-art approaches, the new method offers lower errors in both location and orientation registration. Our method offers highly automated and accurate IVUS pullback registration and can be employed in IVUS-based studies of coronary disease progression, enabling more focal studies of coronary plaque development and transition of vulnerability.

When segmenting intraretinal layers from multiple optical coherence tomography (OCT) images forming a mosaic or a set of repeated scans, it is attractive to exploit the additional information from the overlapping areas rather than discarding it as redundant, especially in low contrast and noisy images. However, it is currently not clear how to effectively combine the multiple information sources available in the areas of overlap. In this paper, we propose a novel graph-theoretic method for multi-surface multi-field co-segmentation of intraretinal layers, assuring consistent segmentation of the fields across the overlapped areas. After 2-D en-face alignment, all the fields are segmented simultaneously, imposing a priori soft interfield-intrasurface constraints for each pair of overlapping fields. The constraints penalize deviations from the expected surface height differences, taken to be the depth-axis shifts that produce the maximum cross-correlation of pairwise-overlapped areas. The method's accuracy and reproducibility are evaluated qualitatively and quantitatively on 212 OCT images (20 nine-field, 32 single-field acquisitions) from 26 patients with glaucoma. Qualitatively, the obtained thickness maps show no stitching artifacts, compared to pronounced stitches when the fields are segmented independently. Quantitatively, two ophthalmologists manually traced four intraretinal layers on 10 patients, and the average error ( 4.58 ±1.46 μm) was comparable to the average difference between the observers ( 5.86±1.72 μm). Furthermore, we show the benefit of the proposed approach in co-segmenting longitudinal scans. As opposed to segmenting layers in each of the fields independently, the proposed co-segmentation method obtains consistent segmentations across the overlapped areas, producing accurate, reproducible, and artifact-free results.

Automated reconstruction of the cortical surface is one of the most challenging problems in the analysis of human brain magnetic resonance imaging (MRI). A desirable segmentation must be both spatially and topologically accurate, as well as robust and computationally efficient. We propose a novel algorithm, LOGISMOS-B, based on probabilistic tissue classification, generalized gradient vector flows and the LOGISMOS graph segmentation framework. Quantitative results on MRI datasets from both healthy subjects and multiple sclerosis patients using a total of 16,800 manually placed landmarks illustrate the excellent performance of our algorithm with respect to spatial accuracy. Remarkably, the average signed error was only 0.084 mm for the white matter and 0.008 mm for the gray matter, even in the presence of multiple sclerosis lesions. Statistical comparison shows that LOGISMOS-B produces a significantly more accurate cortical reconstruction than FreeSurfer, the current state-of-the-art approach (p << 0.001). Furthermore, LOGISMOS-B enjoys a run time that is less than a third of that of FreeSurfer, which is both substantial, considering the latter takes 10 h/subject on average, and a statistically significant speedup.

The correct segmentation of blood vessels in optical coherence tomography (OCT) images is an important requirement for better diagnosis of many retinal diseases. Although OCT blood vessel segmentation is often performed by applying vessel detection methods on 2D projection of OCT datasets, some papers investigate the vessel segmentation on OCT slices. The presence of shadows in outer retinal layers is established as the main factor for vessel localization; however, the shadow information fails to localize many important blood vessels. The proposed method is based on anatomical changes of Retinal Nerve Fiber Layer (RNFL) in presence of vessels. In this paper we find the thickening of RNFL by applying a layer segmentation algorithm on OCT slices and combine this information with shadow localization. Furthermore, a vessel detection method based on curvelet transform is also applied on 2D projection of OCTs to be added to localized vessels from OCTs. The results show that combination of vessel detection on 2D projection with vessel localization on OCTs can improve the accuracy up to 0.96 which is promisingly higher than older methods.

Optical coherence tomography (OCT) is known to be one of the powerful and noninvasive methods in retinal imaging. OCT uses retroreflected light to provide micron-resolution, cross-sectional scans of biological tissues. In contrast to OCT technology development, which has been a field of active research since 1991, OCT image segmentation has only been fully explored during the last decade. In this paper, we introduce a fast segmentation method based on a new kind of spectral graph theory named diffusion maps. The research is performed on spectral domain OCT images depicting normal macular appearance. In contrast to our recent methods of graph based OCT image segmentation, the presented approach does not require edge-based image information and rather relies on regional image texture. Consequently, the proposed method demonstrates robustness in situations of low image contrast or poor layer-to-layer image gradients. This method is tested on thirteen 3D macular SD-OCT images obtained from eyes without pathologies with Topcon 3D OCT-1000 imaging system (with a size of 650 × 512 × 128 voxels and a voxel resolution of 4.81 × 13.67 × 24.41 μm3). The mean unsigned and signed border positioning errors (mean ± SD) was 8.52±3.13 and -4.61±3.35 micrometers, respectively. The average computation time of the proposed algorithms (implemented with MATLAB) was 12 seconds per 2D slice.

Identification of both the luminal and the wall areas of the bronchial tree structure from volumetric X-ray computed tomography (CT) data sets is of critical importance in distinguishing important phenotypes within numerous major lung diseases including chronic obstructive pulmonary diseases (COPD) and asthma. However, accurate assessment of the inner and outer airway wall surfaces of a complete 3-D tree structure is difficult due to their complex nature, particularly around the branch areas. In this paper, we extend a graph search based technique (LOGISMOS) to simultaneously identify multiple inter-related surfaces of branching airway trees. We first perform a presegmentation of the input 3-D image to obtain basic information about the tree topology. The presegmented image is resampled along judiciously determined paths to produce a set of vectors of voxels (called voxel columns). The resampling process utilizes medial axes to ensure that voxel columns of appropriate lengths and directions are used to capture the object surfaces without interference. A geometric graph is constructed whose edges connect voxels in the resampled voxel columns and enforce validity of the smoothness and separation constraints on the sought surfaces. Cost functions with directional information are employed to distinguish inner and outer walls. The assessment of wall thickness measurement on a CT-scanned double-wall physical phantom (patterned after an in vivo imaged human airway tree) achieved highly accurate results on the entire 3-D tree. The observed mean signed error of wall thickness ranged from -0.09 ±0.24 mm to 0.07 ±0.23 mm in bifurcating/nonbifurcating areas. The mean unsigned errors were 0.16±0.12 mm to 0.20±0.11 mm. When the airway wall surface was partitioned into meaningful subregions, the airway wall thickness accuracy was the same in most tested bifurcation/nonbifurcation and carina/noncarina regions (p=NS). Once validated on phantoms, our method was applied to human in vivo volumetric CT data to demonstrate relationships of airway wall thickness as a function of luminal dimension and airway tree generation. Wall thickness differences between the bifurcation/nonbifurcation regions were statistically significant (p < 0.05) for tree generations 6, 7, 8, and 9. In carina/noncarina regions, the wall thickness was statistically different in generations 1, 4, 5, 6, 7, and 8.

Segmentation of multiple surfaces in medical images is a challenging problem, further complicated by the frequent presence of weak boundary evidence, large object deformations, and mutual influence between adjacent objects. This paper reports a novel approach to multi-object segmentation that incorporates both shape and context prior knowledge in a 3-D graph-theoretic framework to help overcome the stated challenges. We employ an arc-based graph representation to incorporate a wide spectrum of prior information through pair-wise energy terms. In particular, a shape-prior term is used to penalize local shape changes and a context-prior term is used to penalize local surface-distance changes from a model of the expected shape and surface distances, respectively. The globally optimal solution for multiple surfaces is obtained by computing a maximum flow in a low-order polynomial time. The proposed method was validated on intraretinal layer segmentation of optical coherence tomography images and demonstrated statistically significant improvement of segmentation accuracy compared to our earlier graph-search method that was not utilizing shape and context priors. The mean unsigned surface positioning errors obtained by the conventional graph-search approach (6.30 ±1.58 μ m) was improved to 5.14±0.99 μ m when employing our new method with shape and context priors.

A new graph-based approach for segmentation of luminal and external elastic lamina (EEL) surface of coronary vessels in gated 20 MHz intravascular ultrasound (IVUS) image sequences (volumes) is presented. The approach consists of a fully automated segmentation stage ("new automated" or NA) and a user-guided computer-aided refinement ("new refinement" or NR) stage. Both approaches are based on the LOGISMOS approach for simultaneous dual-surface graph-based segmentation. This combination allows the user to efficiently combine general information about IVUS image appearance and case-specific IVUS morphology and therefore deal with frequently occurring issues like calcified plaque-causing signal shadowing-and imaging artifacts. The automated segmentation stage starts with pre-segmenting the lumen to automatically define the lumen centerline, which is used to transform the segmentation task into a LOGISMOS-family graph optimization problem. Following the automated segmentation, the user can inspect the result and correct local or regional segmentation inaccuracies by (iteratively) providing approximate clues regarding the location of the desired surface locations. This expert information is utilized to modify the previously calculated cost functions, locally re-optimizing the underlying modified graph without a need to start the new optimization from scratch. Validation of our method was performed on 41 gated 20 MHz IVUS data sets for which an expert-defined independent standard was available. Resulting from the automated stage of the approach (NA), the mean and standard deviation of the root mean square area errors for the luminal and external elastic lamina surfaces were 1.12 ±0.67 mm (2) and 2.35 ±1.61 mm (2) , respectively. Following the refinement stage (NR), the root mean square area errors significantly decreased to 0.82 ±0.44 mm (2) and 1.17 ±0.65 mm (2) for the same surfaces, respectively ( for both surfaces). The approach is delivering a previously unachievable speed of obtaining clinically relevant segmentations compared to the current approaches of automated segmentation followed by manual editing.

An automated method is reported for segmenting 3-D fluid-associated abnormalities in the retina, so-called symptomatic exudate-associated derangements (SEAD), from 3-D OCT retinal images of subjects suffering from exudative age-related macular degeneration. In the first stage of a two-stage approach, retinal layers are segmented, candidate SEAD regions identified, and the retinal OCT image is flattened using a candidate-SEAD aware approach. In the second stage, a probability constrained combined graph search-graph cut method refines the candidate SEADs by integrating the candidate volumes into the graph cut cost function as probability constraints. The proposed method was evaluated on 15 spectral domain OCT images from 15 subjects undergoing intravitreal anti-VEGF injection treatment. Leave-one-out evaluation resulted in a true positive volume fraction (TPVF), false positive volume fraction (FPVF) and relative volume difference ratio (RVDR) of 86.5%, 1.7%, and 12.8%, respectively. The new graph cut-graph search method significantly outperformed both the traditional graph cut and traditional graph search approaches (p < 0.01, p < 0.04) and has the potential to improve clinical management of patients with choroidal neovascularization due to exudative age-related macular degeneration.

In this paper, we propose a novel method to reduce the magnitude of 4D CT artifacts by stitching two images with a data-driven regularization constrain, which helps preserve the local anatomy structures. Our method first computes an interface seam for the stitching in the overlapping region of the first image, which passes through the "smoothest" region, to reduce the structure complexity along the stitching interface. Then, we compute the displacements of the seam by matching the corresponding interface seam in the second image. We use sparse 3D features as the structure cues to guide the seam matching, in which a regularization term is incorporated to keep the structure consistency. The energy function is minimized by solving a multiple-label problem in Markov Random Fields with an anatomical structure preserving regularization term. The displacements are propagated to the rest of second image and the two image are stitched along the interface seams based on the computed displacement field. The method was tested on both simulated data and clinical 4D CT images. The experiments on simulated data demonstrated that the proposed method was able to reduce the landmark distance error on average from 2.9 mm to 1.3 mm, outperforming the registration-based method by about 55%. For clinical 4D CT image data, the image quality was evaluated by three medical experts, and all identified much fewer artifacts from the resulting images by our method than from those by the compared method.

This is an introduction to a special session of ICASSP devoted to signal processing techniques in medical imaging and image analysis that consists of this introduction and 5 research presentations, each addressing one aspect of the medical imaging field in which signal processing plays an irreplaceable role. The topics cover a broad spectrum of medical imaging problems from image acquisition to image analysis to population-based anatomical modeling. The focus is on technical aspects of the problem and proposed/developed solutions with sufficient emphasis given to the introducing the biomedical importance of the problem as well as its signal processing relevance.

Efficient segmentation of globally optimal surfaces in volumetric images is a central problem in many medical image analysis applications. Intraclass variance has been successfully utilized for object segmentation, for instance, in the Chan-Vese model, especially for images without prominent edges. In this paper, we study the optimization problem of detecting a region (volume) between two coupled smooth surfaces by minimizing the intraclass variance using an efficient polynomial-time algorithm. Our algorithm is based on the shape probing technique in computational geometry and computes a sequence of minimum-cost closed sets in a derived parametric graph. The method has been validated on computer-synthetic volumetric images and in X-ray CT-scanned datasets of plexiglas tubes of known sizes. Its applicability to clinical data sets was also demonstrated. In all cases, the approach yielded highly accurate results. We believe that the developed technique is of interest on its own. We expect that it can shed some light on solving other important optimization problems arising in medical imaging. Furthermore, we report an approximation algorithm which runs much faster than the exact algorithm while yielding highly comparable segmentation accuracy.

This paper proposes an algorithm to measure the width of retinal vessels in fundus photographs using graph-based algorithm to segment both vessel edges simultaneously. First, the simultaneous two-boundary segmentation problem is modeled as a two-slice, 3-D surface segmentation problem, which is further converted into the problem of computing a minimum closed set in a node-weighted graph. An initial segmentation is generated from a vessel probability image. We use the REVIEW database to evaluate diameter measurement performance. The algorithm is robust and estimates the vessel width with subpixel accuracy. The method is used to explore the relationship between the average vessel width and the distance from the optic disc in 600 subjects.

Multiple surface searching with only image intensity information is a difficult job in the presence of high noise and weak edges. We present in this paper a novel method for globally optimal multi-surface searching with a shape prior represented by convex pairwise energies. A 3-D graph-theoretic framework is employed. An arc-weighted graph is constructed based on a shape model built from training datasets. A wide spectrum of constraints is then incorporated. The shape prior term penalizes the local topological change from the original shape model. The globally optimal solution for multiple surfaces can be obtained by computing a maximum flow in low-order polynomial time. Compared with other graph-based methods, our approach provides more local and flexible control of the shape. We also prove that our algorithm can handle the detection of multiple crossing surfaces with no shared voxels. Our method was applied to several application problems, including medical image segmentation, scenic image segmentation, and image resizing. Compared with results without using shape prior information, our improvement was quite impressive, demonstrating the promise of our method.

Glaucoma is the second leading ocular disease causing blindness due to gradual damage to the optic nerve and resultant visual field loss. Segmentations of the optic disc cup and neuroretinal rim can provide important parameters for detecting and tracking this disease. The purpose of this study is to describe and evaluate a method that can automatically segment the optic disc cup and rim in spectral-domain 3-D OCT (SD-OCT) volumes. Four intraretinal surfaces were segmented using a fast multiscale 3-D graph search algorithm. After surface segmentation, the retina in each 3-D OCT scan was flattened to ensure a consistent optic nerve head shape. A set of 15 features, derived from the segmented intraretinal surfaces and voxel intensities in the SD-OCT volume, were used to train a classifier that can determine which A-scans in the OCT volume belong to the background, optic disc cup and rim. Finally, prior knowledge about the shapes of the cup and rim was incorporated into the system using a convex hull-based approach. Two glaucoma experts annotated the cup and rim area using planimetry, and the annotations of the first expert were used as the reference standard. A leave-one-subject-out experiment on 27 optic nerve head-centered OCT volumes (14 right eye scans and 13 left eye scans from 14 patients) was performed. Two different types of classification methods were compared, and experimental results showed that the best performing method had an unsigned error for the optic disc cup of 2.52+/-0.87 pixels (0.076+/-0.026 mm) and for the neuroretinal rim of 2.04+/-0.86 pixels (0.061+/-0.026 mm). The interobserver variability as indicated by the unsigned border positioning difference between the second expert observer and the reference standard was 2.54+/-1.03 pixels (0.076+/-0.031 mm for the optic disc cup and 2.14+/-0.80 pixels (0.064+/-0.024 mm for the neuroretinal rim. The unsigned error of the best performing method was not significantly different (p > 0.2) from the interobserver variability.

Optical coherence tomography (OCT) is becoming one of the most important modalities for the noninvasive assessment of retinal eye diseases. As the number of acquired OCT volumes increases, automating the OCT image analysis is becoming increasingly relevant. In this paper, a method for automated characterization of the normal macular appearance in spectral domain OCT (SD-OCT) volumes is reported together with a general approach for local retinal abnormality detection. Ten intraretinal layers are first automatically segmented and the 3-D image dataset flattened to remove motion-based artifacts. From the flattened OCT data, 23 features are extracted in each layer locally to characterize texture and thickness properties across the macula. The normal ranges of layer-specific feature variations have been derived from 13 SD-OCT volumes depicting normal retinas. Abnormalities are then detected by classifying the local differences between the normal appearance and the retinal measures in question. This approach was applied to determine footprints of fluid-filled regions--SEADs (Symptomatic Exudate-Associated Derangements)--in 78 SD-OCT volumes from 23 repeatedly imaged patients with choroidal neovascularization (CNV), intra-, and sub-retinal fluid and pigment epithelial detachment. The automated SEAD footprint detection method was validated against an independent standard obtained using an interactive 3-D SEAD segmentation approach. An area under the receiver-operating characteristic curve of 0.961 +/- 0.012 was obtained for the classification of vertical, cross-layer, macular columns. A study performed on 12 pairs of OCT volumes obtained from the same eye on the same day shows that the repeatability of the automated method is comparable to that of the human experts. This work demonstrates that useful 3-D textural information can be extracted from SD-OCT scans and--together with an anatomical atlas of normal retinas--can be used for clinically important applications.

A novel multiscale topomorphologic approach for opening of two isointensity objects fused at different locations and scales is presented and applied to separating arterial and venous trees in 3-D pulmonary multidetector X-ray computed tomography (CT) images. Initialized with seeds, the two isointensity objects (arteries and veins) grow iteratively while maintaining their spatial exclusiveness and eventually form two mutually disjoint objects at convergence. The method is intended to solve the following two fundamental challenges: how to find local size of morphological operators and how to trace continuity of locally separated regions. These challenges are met by combining fuzzy distance transform (FDT), a morphologic feature with a topologic fuzzy connectivity, and a new morphological reconstruction step to iteratively open finer and finer details starting at large scales and progressing toward smaller scales. The method employs efficient user intervention at locations where local morphological separability assumption does not hold due to imaging ambiguities or any other reason. The approach has been validated on mathematically generated tubular objects and applied to clinical pulmonary noncontrast CT data for separating arteries and veins. The tradeoff between accuracy and the required user intervention for the method has been quantitatively examined by comparing with manual outlining. The experimental study, based on a blind seed selection strategy, has demonstrated that above 95% accuracy may be achieved using 25-40 seeds for each of arteries and veins. Our method is very promising for semiautomated separation of arteries and veins in pulmonary CT images even when there is no object-specific intensity variation at conjoining locations.

A novel method for simultaneous segmentation of multiple interacting surfaces belonging to multiple interacting objects, called LOGISMOS (layered optimal graph image segmentation of multiple objects and surfaces), is reported. The approach is based on the algorithmic incorporation of multiple spatial inter-relationships in a single n-dimensional graph, followed by graph optimization that yields a globally optimal solution. The LOGISMOS method's utility and performance are demonstrated on a bone and cartilage segmentation task in the human knee joint. Although trained on only a relatively small number of nine example images, this system achieved good performance. Judged by dice similarity coefficients (DSC) using a leave-one-out test, DSC values of 0.84 ± 0.04, 0.80 ± 0.04 and 0.80 ± 0.04 were obtained for the femoral, tibial, and patellar cartilage regions, respectively. These are excellent DSC values, considering the narrow-sheet character of the cartilage regions. Similarly, low signed mean cartilage thickness errors were obtained when compared to a manually-traced independent standard in 60 randomly selected 3-D MR image datasets from the Osteoarthritis Initiative database-0.11 ± 0.24, 0.05 ± 0.23, and 0.03 ± 0.17 mm for the femoral, tibial, and patellar cartilage thickness, respectively. The average signed surface positioning errors for the six detected surfaces ranged from 0.04 ± 0.12 mm to 0.16 ± 0.22 mm. The reported LOGISMOS framework provides robust and accurate segmentation of the knee joint bone and cartilage surfaces of the femur, tibia, and patella. As a general segmentation tool, the developed framework can be applied to a broad range of multiobject multisurface segmentation problems.

In this study, a combination of active shape model (ASM) and active appearance model (AAM) was used to segment the left and right ventricles of normal and Tetralogy of Fallot (TOF) hearts on 4-D (3-D+time) MR images. For each ventricle, a 4-D model was first used to achieve robust preliminary segmentation on all cardiac phases simultaneously and a 3-D model was then applied to each phase to improve local accuracy while maintaining the overall robustness of the 4-D segmentation. On 25 normal and 25 TOF hearts, in comparison to the expert traced independent standard, our comprehensive performance assessment showed subvoxel segmentation accuracy, high overlap ratios, good ventricular volume correlations, and small percent volume differences. Following 4-D segmentation, novel quantitative shape and motion features were extracted using shape information, volume-time and dV/dt curves, analyzed and used for disease status classification. Automated discrimination between normal/TOF subjects achieved 90%-100% sensitivity and specificity. The features obtained from TOF hearts show higher variability compared to normal subjects, suggesting their potential use as disease progression indicators. The abnormal shape and motion variations of the TOF hearts were accurately captured by both the segmentation and feature characterization.

Content-aware video/image resizing is of increasing relevance to allow high-quality image and video resizing to be displayed on devices with different resolution. In this paper, we present a novel algorithm to find multiple 3-D surfaces simultaneously with globally optimal solution for video/image resizing. Our algorithm is based on graph theory and it first analyzes the video/image data to define the energy value for each voxel. Then, a 4-D graph is constructed and the costs are assigned according to the energy values. Finally, multiple 3-D surfaces are detected by a global optimization process which can be solved via s-t graph cuts. By removing or inserting these multiple 3-D surfaces, content-aware video/image resizing is achieved. We also have proved that our algorithm can find the globally optimal solution for crossing surfaces problem, in which several surfaces can cross each other. The proposed method is demonstrated on a variety of video/image data and compared to the state of the art in video/image resizing.

With the introduction of spectral-domain optical coherence tomography (OCT), much larger image datasets are routinely acquired compared to what was possible using the previous generation of time-domain OCT. Thus, the need for 3-D segmentation methods for processing such data is becoming increasingly important. We report a graph-theoretic segmentation method for the simultaneous segmentation of multiple 3-D surfaces that is guaranteed to be optimal with respect to the cost function and that is directly applicable to the segmentation of 3-D spectral OCT image data. We present two extensions to the general layered graph segmentation method: the ability to incorporate varying feasibility constraints and the ability to incorporate true regional information. Appropriate feasibility constraints and cost functions were learned from a training set of 13 spectral-domain OCT images from 13 subjects. After training, our approach was tested on a test set of 28 images from 14 subjects. An overall mean unsigned border positioning error of 5.69+/-2.41 microm was achieved when segmenting seven surfaces (six layers) and using the average of the manual tracings of two ophthalmologists as the reference standard. This result is very comparable to the measured interobserver variability of 5.71+/-1.98 microm.

Efficient segmentation of globally optimal surfaces in volumetric images is a central problem in many medical image analysis applications. Intra-class variance has been successfully utilized, for instance, in the Chan-Vese model especially for images without prominent edges. In this paper, we study the optimization problem of detecting a region (volume) between two coupled smooth surfaces by minimizing the intra-class variance using an efficient polynomial-time algorithm. Our algorithm is based on the shape probing technique in computational geometry and computes a sequence of minimum-cost closed sets in a derived parametric graph. The method has been validated on computer-synthetic volumetric images and in X-ray CT-scanned datasets of plexiglas tubes of known sizes. Its applicability to clinical data sets was demonstrated in human CT image data. The achieved results were highly accurate with mean signed surface positioning errors of the inner and outer walls of the tubes of +0.013mm and 0.012mm, respectively, given a voxel size of 0.39 × 0.39 × 0.6mm3. Comparing with the original Chan-Vese method [8], our algorithm expressed higher robustness. With its polynomialtime efficiency, our algorithm is ready to be extended to higher-dimensional image segmentation. In addition, the developed technique is of its own interest. We expect that it can shed some light on solving other important optimization problems arising in computer vision. To the best of our knowledge, the shape probing technique is for the first time introduced into the field of computer vision.

Current techniques for segmenting macular optical coherence tomography (OCT) images have been 2-D in nature. Furthermore, commercially available OCT systems have only focused on segmenting a single layer of the retina, even though each intraretinal layer may be affected differently by disease. We report an automated approach for segmenting (anisotropic) 3-D macular OCT scans into five layers. Each macular OCT dataset consisted of six linear radial scans centered at the fovea. The six surfaces defining the five layers were identified on each 3-D composite image by transforming the segmentation task into that of finding a minimum-cost closed set in a geometric graph constructed from edge/regional information and a priori determined surface smoothness and interaction constraints. The method was applied to the macular OCT scans of 12 patients (24 3-D composite image datasets) with unilateral anterior ischemic optic neuropathy (AION). Using the average of three experts' tracings as a reference standard resulted in an overall mean unsigned border positioning error of 6.1 +/- 2.9 microm, a result comparable to the interobserver variability (6.9 +/- 3.3 microm). Our quantitative analysis of the automated segmentation results from AION subject data revealed that the inner retinal layer thickness for the affected eye was 24.1 microm (21%) smaller on average than for the unaffected eye (p < 0.001), supporting the need for segmenting the layers separately.

Automated and accurate segmentation of the aorta in 4D (3D+time) cardiovascular magnetic resonance (MR) image data is important for early detection of congenital aortic disease leading to aortic aneurysms and dissections. An automated 4D segmentation method is reported in this study. Our automated segmentation method combines level-set and optimal surface segmentation algorithms so that the final aortic surfaces in n cardiac phases are determined in a single optimization process. 4D MR image data sets acquired from 21 normal and connective tissue disorder subjects were used to evaluate the performance of our method. The automated 4D segmentation results produced accurate aortic surfaces in 16 cardiac phases, covering the aorta from the left-ventricular outflow tract to the diaphragm, yielding subvoxel accuracy with signed surface positioning errors of-0.07±1.16 voxel (-0.10±2.05 mm). 1

Efficient segmentation of globally optimal surfaces representing object boundaries in volumetric data sets is important and challenging in many medical image analysis applications. We have developed an optimal surface detection method capable of simultaneously detecting multiple interacting surfaces, in which the optimality is controlled by the cost functions designed for individual surfaces and by several geometric constraints defining the surface smoothness and interrelations. The method solves the surface segmentation problem by transforming it into computing a minimum s-t cut in a derived arc-weighted directed graph. The proposed algorithm has a low-order polynomial time complexity and is computationally efficient. It has been extensively validated on more than 300 computer-synthetic volumetric images, 72 CT-scanned data sets of different-sized plexiglas tubes, and tens of medical images spanning various imaging modalities. In all cases, the approach yielded highly accurate results. Our approach can be readily extended to higher-dimensional image segmentation.

This paper describes a new approach to the automated segmentation of X-ray left ventricular (LV) angiograms, based on active appearance models (AAMs) and dynamic programming. A coupling of shape and texture information between the end-diastolic (ED) and end-systolic (ES) frame was achieved by constructing a multiview AAM. Over-constraining of the model was compensated for by employing dynamic programming, integrating both intensity and motion features in the cost function. Two applications are compared: a semi-automatic method with manual model initialization, and a fully automatic algorithm. The first proved to be highly robust and accurate, demonstrating high clinical relevance. Based on experiments involving 70 patient data sets, the algorithm's success rate was 100% for ED and 99% for ES, with average unsigned border positioning errors of 0.68 mm for ED and 1.45 mm for ES. Calculated volumes were accurate and unbiased. The fully automatic algorithm, with intrinsically less user interaction was less robust, but showed a high potential, mostly due to a controlled gradient descent in updating the model parameters. The success rate of the fully automatic method was 91% for ED and 83% for ES, with average unsigned border positioning errors of 0.79 mm for ED and 1.55 mm for ES.

Our goal is to enhance the ability to differentiate normal lung from subtle pathologies via multidetector row CT (MDCT) by extending a two-dimensional (2-D) texturebased tissue classification [adaptive multiple feature method (AMFM)] to use three-dimensional (3-D) texture features. We performed MDCT on 34 humans and classified volumes of interest (VOIs) in the MDCT images into five categories: EC, emphysema in severe chronic obstructive pulmonary disease (COPD); MC, mild emphysema in mild COPD; NC, normal appearing lung in mild COPD; NN, normal appearing lung in normal nonsmokers; and NS, normal appearing lung in normal smokers. COPD severity was based upon pulmonary function tests (PFTs). Airways and vessels were excluded from VOIs; 24 3-D texture features were calculated; and a Bayesian classifier was used for discrimination. A leave-one-out method was employed for validation. Sensitivity of the four-class classification in the form of 3-D/2-D was: EC: 85%/71%, MC: 90%/82%; NC: 88%/50%; NN: 100%/60%. Sensitivity and specificity for NN using a two-class classification of NN and NS in the form of 3-D/2-D were: 99%/72% and 100%/75%, respectively. We conclude that 3-D AMFM analysis of lung parenchyma improves discrimination compared to 2-D AMFM of the same VOIs. Furthermore, our results suggest that the 3-D AMFM may provide a means of discriminating subtle differences between smokers and nonsmokers both with normal PFTs.

We introduce a new type of data for classification of regional segments of myocardium. We have analyzed strain measurements taken throughout the cardiac cycle from the echocardiograms of pigs. Classifications by both principal component analysis (PCA) and by neural network (NN) are combined for a data mining operation. Differences in strain waveforms between normal and diseased myocardium may further elucidate the corresponding changes in physiology. Altered functioning of the heart muscle is reflected by strain, and objective computer analysis should aid in the diagnosis of ischemia. We hypothesize that the entire strain waveform over one heart cycle can be classified to functionally determine whether or not a myocardial region is perfused.

Active appearance models (AAMs) have been successfully used for a variety of segmentation tasks in medical image analysis. However, gross disturbances of objects can occur in routine clinical setting caused by pathological changes or medical interventions. This poses a problem for AAM-based segmentation, since the method is inherently not robust. In this paper, a novel robust AAM (RAAM) matching algorithm is presented. Compared to previous approaches, no assumptions are made regarding the kind of gray-value disturbance and/or the expected magnitude of residuals during matching. The method consists of two main stages. First, initial residuals are analyzed by means of a mean-shift-based mode detection step. Second, an objective function is utilized for the selection of a mode combination not representing the gross outliers. We demonstrate the robustness of the method in a variety of examples with different noise conditions. The RAAM performance is quantitatively demonstrated in two substantially different applications, diaphragm segmentation and rheumatoid arthritis assessment. In all cases, the robust method shows an excellent behavior, with the new method tolerating up to 50% object area covered by gross gray-level disturbances.

The segmentation of the human airway tree from volumetric computed tomography (CT) images builds an important step for many clinical applications and for physiological studies. Previously proposed algorithms suffer from one or several problems: leaking into the surrounding lung parenchyma, the need for the user to manually adjust parameters, excessive runtime. Low-dose CT scans are increasingly utilized in lung screening studies, but segmenting them with traditional airway segmentation algorithms often yields less than satisfying results. In this paper, a new airway segmentation method based on fuzzy connectivity is presented. Small adaptive regions of interest are used that follow the airway branches as they are segmented. This has several advantages. It makes it possible to detect leaks early and avoid them, the segmentation algorithm can automatically adapt to changing image parameters, and the computing time is kept within moderate values. The new method is robust in the sense that it works on various types of scans (low-dose and regular dose, normal subjects and diseased subjects) without the need for the user to manually adjust any parameters. Comparison with a commonly used region-grow segmentation algorithm shows that the newly proposed method retrieves a significantly higher count of airway branches. A method that conducts accurate cross-sectional airway measurements on airways is presented as an additional processing step. Measurements are conducted in the original gray-level volume. Validation on a phantom shows that subvoxel accuracy is achieved for all airway sizes and airway orientations.

Matching of corresponding branchpoints between two human airway trees, as well as assigning anatomical names to the segments and branchpoints of the human airway tree, are of significant interest for clinical applications and physiological studies. In the past, these tasks were often performed manually due to the lack of automated algorithms that can tolerate false branches and anatomical variability typical for in vivo trees. In this paper, we present algorithms that perform both matching of branchpoints and anatomical labeling of in vivo trees without any human intervention and within a short computing time. No hand-pruning of false branches is required. The results from the automated methods show a high degree of accuracy when validated against reference data provided by human experts. 92.9% of the verifiable branchpoint matches found by the computer agree with experts' results. For anatomical labeling, 97.1% of the automatically assigned segment labels were found to be correct.

A novel approach for platform-independent virtual endoscopy in human coronary arteries is presented in this paper. It incorporates previously developed and validated methodology for multimodality fusion of two X-ray angiographic images with pullback data from intravascular ultrasound (IVUS). These modalities pose inherently different challenges than those present in many tomographic modalities that provide parallel slices. The fusion process results in a three- or four-dimensional (3-D/4-D) model of a coronary artery, specifically of its lumen/plaque and media/adventitia surfaces. The model is used for comprehensive quantitative hemodynamic, morphologic, and functional analyses. The resulting quantitative indexes are then used to supplement the model. Platform-independent visualization is achieved through the use of the ISO/IEC-standardized Virtual Reality Modeling Language (VRML). The visualization includes an endoscopic fly-through animation that enables the user to interactively select vessel location and fly-through speed, as well as to display image pixel data or quantification results in 3-D. The presented VRML virtual-endoscopy system is used in research studies of coronary atherosclerosis development, quantitative assessment of coronary morphology and function, and vascular interventions.

A novel extension of active appearance models (AAMs) for automated border detection in echocardiographic image sequences is reported. The active appearance motion model (AAMM) technique allows fully automated robust and time-continuous delineation of left ventricular (LV) endocardial contours over the full heart cycle with good results. Nonlinear intensity normalization was developed and employed to accommodate ultrasound-specific intensity distributions. The method was trained and tested on 16-frame phase-normalized transthoracic four-chamber sequences of 129 unselected infarct patients, split randomly into a training set (n = 65) and a test set (n = 64). Borders were compared to expert drawn endocardial contours. On the test set, fully automated AAMM performed well in 97% of the cases (average distance between manual and automatic landmark points was 3.3 mm, comparable to human interobserver variabilities). The ultrasound-specific intensity normalization proved to be of great value for good results in echocardiograms. The AAMM was significantly more accurate than an equivalent set of two-dimensional AAMs.

A model-based method for three-dimensional image segmentation was developed and its performance assessed in segmentation of volumetric cardiac magnetic resonance (MR) images and echocardiographic temporal image sequences. Comprehensive design of a three-dimensional (3-D) active appearance model (AAM) is reported for the first time as an involved extension of the AAM framework introduced by Cootes et al. The model's behavior is learned from manually traced segmentation examples during an automated training stage. Information about shape and image appearance of the cardiac structures is contained in a single model. This ensures a spatially and/or temporally consistent segmentation of three-dimensional cardiac images. The clinical potential of the 3-D AAM is demonstrated in short-axis cardiac MR images and four-chamber echocardiographic sequences. The method's performance was assessed by comparison with manually identified independent standards in 56 clinical MR and 64 clinical echo image sequences. The AAM method showed good agreement with the independent standard using quantitative indexes of border positioning errors, endo- and epicardial volumes, and left ventricular mass. In MR, the endocardial volumes, epicardial volumes, and left ventricular wall mass correlation coefficients between manual and AAM were R2 = 0.94, 0.97, 0.82, respectively. For echocardiographic analysis, the area correlation was R2 = 0.79. The AAM method shows high promise for successful application to MR and echocardiographic image analysis in a clinical setting.

Early detection of cardiovascular disease would allow timely institution of preventive measures. Arterial endothelium play a primary role in processes leading to the development of atherosclerotic plaque and cardiovascular disease in general. Determination of flow-mediated dilatation (FMD) of brachial arteries from B-mode ultrasound image sequences offers a noninvasive surrogate index of endothelial function. A highly automated method for analysis of brachial ultrasound image sequences is reported and its performance assessed. The method overcomes the variability of brachial ultrasound images across subjects by incorporating machine learning and quality control steps. The automated method outperformed conventional manual analysis by providing a decreased analysis bias, increased reproducibility, and improved measurement accuracy. Consequently, it decreases inter- and intraobserver as well interinstitution variability. The method has been employed in a number of population studies with thousands of subjects analyzed.

A novel approach to three-dimensional segmentation, parametric surface representation, and interactive surface modification is reported. The combination of shape-based and appearance-based volumetric segmentation, parametric representation of 3D surfaces, and their interactive modification and editing forms a very powerful paradigm for a variety of volumetric segmentation tasks. Performance assessment of the method for automated segmentation of diaphragm surfaces in volumetric 3D CT images is ongoing.

A fully automated approach to segmentation of the left and right cardiac ventricles from magnetic resonance (MR) images is reported. A novel multistage hybrid appearance model methodology is presented in which a hybrid active shape model/active appearance model (AAM) stage helps avoid local minima of the matching function. This yields an overall more favorable matching result. An automated initialization method is introduced making the approach fully automated. Our method was trained in a set of 102 MR images and tested in a separate set of 60 images. In all testing cases, the matching resulted in a visually plausible and accurate mapping of the model to the image data. Average signed border positioning errors did not exceed 0.3 mm in any of the three determined contours-left-ventricular (LV) epicardium, LV and right-ventricular (RV) endocardium. The area measurements derived from the three contours correlated well with the independent standard (r = 0.96, 0.96, 0.90), with slopes and intercepts of the regression lines close to one and zero, respectively. Testing the reproducibility of the method demonstrated an unbiased performance with small range of error as assessed via Bland-Altman statistic. In direct border positioning error comparison, the multistage method significantly outperformed the conventional AAM (p < 0.001). The developed method promises to facilitate fully automated quantitative analysis of LV and RV morphology and function in clinical setting.

A highly automated method for the identification and quantization of maximum blood velocity curves from Doppler ultrasound flow diagrams is presented. The method uses an image processing scheme to analyze video-recorded image sequences of flow diagrams. The sequences are acquired, a sequence of images relating to chronological cardiac cycles is extracted, and a maximum blood velocity envelope is determined and quantified. The results are verified against hand-traced reference curves. Excellent correlation of r = 0.99 is achieved.

This paper provides methodology for fully automated model-based image segmentation. All information necessary to perform image segmentation is automatically derived from a training set that is presented in a form of segmentation examples. The training set is used to construct two models representing the objects--shape model and border appearance model. A two-step approach to image segmentation is reported. In the first step, an approximate location of the object of interest is determined. In the second step, accurate border segmentation is performed. The shape-variant Hough transform method was developed that provides robust object localization automatically. It finds objects of arbitrary shape, rotation, or scaling and can handle object variability. The border appearance model was developed to automatically design cost functions that can be used in the segmentation criteria of edge-based segmentation methods. Our method was tested in five different segmentation tasks that included 489 objects to be segmented. The final segmentation was compared to manually defined borders with good results [rms errors in pixels: 1.2 (cerebellum), 1.1 (corpus callosum), 1.5 (vertebrae), 1.4 (epicardial), and 1.6 (endocardial) borders]. Two major problems of the state-of-the-art edge-based image segmentation algorithms were addressed: strong dependency on a close-to-target initialization, and necessity for manual redesign of segmentation criteria whenever new segmentation problem is encountered.

In the rapidly evolving field of intravascular ultrasound (IVUS), the assessment of vessel morphology still lacks a geometrically correct three-dimensional (3-D) reconstruction. The IVUS frames are usually stacked up to form a straight vessel, neglecting curvature and the axial twisting of the catheter during the pullback. Our method combines the information about vessel cross-sections obtained from IVUS with the information about the vessel geometry derived from biplane angiography. First, the catheter path is reconstructed from its biplane projections, resulting in a spatial model. The locations of the IVUS frames are determined and their orientations relative to each other are calculated using a discrete approximation of the Frenet-Serret formulas known from differential geometry. The absolute orientation of the frame set is established, utilizing the imaging catheter itself as an artificial landmark. The IVUS images are segmented, using our previously developed algorithm. The fusion approach has been extensively validated in computer simulations, phantoms, and cadaveric pig hearts.

This paper reports a novel method for fully automated segmentation that is based on description of shape and its variation using point distribution models (PDM's). An improvement of the active shape procedure introduced by Cootes and Taylor to find new examples of previously learned shapes using PDM's is presented. The new method for segmentation and interpretation of deep neuroanatomic structures such as thalamus, putamen, ventricular system, etc. incorporates a priori knowledge about shapes of the neuroanatomic structures to provide their robust segmentation and labeling in magnetic resonance (MR) brain images. The method was trained in eight MR brain images and tested in 19 brain images by comparison to observer-defined independent standards. Neuroanatomic structures in all testing images were successfully identified. Computer-identified and observer-defined neuroanatomic structures agreed well. The average labeling error was 7%+/-3%. Border positioning errors were quite small, with the average border positioning error of 0.8+/-0.1 pixels in 256 x 256 MR images. The presented method was specifically developed for segmentation of neuroanatomic structures in MR brain images. However, it is generally applicable to virtually any task involving deformable shape analysis.

Ovarian ultrasound is an effective tool in infertility treatment. Repeated measurements of the size and shape of follicles over several days are the primary means of evaluation by physicians. Currently, follicle wall segmentation is achieved by manual tracing which is time consuming and susceptible to inter-operator variation. An automated method for follicle wall segmentation is reported that uses a four-step process based on watershed segmentation and knowledge-based graph search algorithm which utilizes priori information about follicle structure for inner and outer wall detection. The automated technique was tested on 36 ultrasonographic images of women's ovaries. Validation against manually traced borders has shown good correlation of manually defined and computer-determined area measurements (R2 = 0.85 - 0.96). The border positioning errors were small: 0.63+/-0.36 mm for inner border and 0.67+/-0.41 mm for outer border detection. The use of watershed segmentation and graph search methods facilitates fast, accurate inner and outer border detection with minimal user-interaction.

Three-dimensional (3-D) analysis of airway trees extracted from computed tomography (CT) image data can provide objective information about lung structure and function. However, manual analysis of 3-D lung CT images is tedious, time consuming and, thus, impractical for routine clinical care. We have previously reported an automated rule-based method for extraction of airway trees from 3-D CT images using a priori knowledge about airway-tree anatomy. Although the method's sensitivity was quite good, its specificity suffered from a large number of falsely detected airways. We present a new approach to airway-tree detection based on fuzzy logic that increases the method's specificity without compromising its sensitivity. The method was validated in 32 CT image slices randomly selected from five volumetric canine electron-beam CT data sets. The fuzzy-logic method significantly outperformed the previously reported rule-based method (p < 0.002).

Intravascular ultrasound (IVUS) imaging permits direct visualization of vascular pathology. It has been used to evaluate lumen and plaque in coronary arteries and its clinical significance for guidance of coronary interventions is increasingly recognized. Conventional manual evaluation is tedious and time-consuming. This paper describes a highly automated approach to segmentation of coronary wall and plaque, and determination of plaque composition in individual IVUS images and pullback image sequences. The determined regions of plaque were classified in one of three classes: soft plaque, hard plaque, or hard plaque shadow. The method's performance was assessed in vitro and in vivo in comparison with observer-defined independent standards. In the analyzed images and image sequences, the mean border positioning error of the wall and plaque borders ranged from 0.13-0.17 mm. Plaque classification correctness was 90%.

Design of a novel three-dimensional (3D) shape-preserving smoothing approach is described. Three-dimensional surfaces are smoothed without shrinkage artifacts typical for many other approaches. Using our new representation of the 3D surface, the process of smoothing performs substantially faster than direct convolution in the spatial domain. The approach shows good smoothing results that preserve 3D shape. The design can be easily extended to n-dimensional filtering.

In coronary vessels smaller than 1 mm in diameter, it is difficult to accurately identify lumen borders using existing border detection techniques. Computer-detected diameters of small coronary vessels are often severely overestimated due to the influence of the imaging system point spread function and the use of an edge operator designed for a broad range of vessel sizes, Computer-detected diameters may be corrected if a calibration curve for the X-ray system is available. Unfortunately, the performance of this postprocessing diameter correction approach is severely limited by the presence of image noise. We report here a new approach that uses a two-stage adaption of edge operator parameters to optimally match the edge operator to the local lumen diameter. In the first stage, approximate lumen diameters are detected using a single edge operator in a half-resolution image. Depending on the approximate lumen size, one of three edge operators is selected for the second full-resolution stage in which left and right coronary borders are simultaneously identified. The method was tested in a set of 72 segments of nine angiographic phantom vessels with diameters ranging from 0.46 to 4.14 mm and in 82 clinical coronary angiograms. Performance of the adaptive simultaneous border detection method was compared to that of a conventional border detection method and to that of a postprocessing diameter correction border detection method. Adaptive border detection yielded significantly improved accuracy in small phantom vessels and across all vessel sizes in comparison to the conventional and postprocessing diameter correction methods (p < 0.001 in all cases). Adaptive simultaneous coronary border detection provides both accurate and robust quantitative analysis of coronary vessels of all sizes.

New sensitive and reliable methods for assessing alterations in regional lung structure and function are critically important for the investigation and treatment of pulmonary diseases. Accurate identification of the airway tree will provide an assessment of airway structure and will provide a means by which multiple volumetric images of the lung at the same lung volume over time can be used to assess regional parenchymal changes. The authors describe a novel rule-based method for the segmentation of airway trees from three-dimensional (3-D) sets of computed tomography (CT) images, and its validation. The presented method takes advantage of a priori anatomical knowledge about pulmonary airway and vascular trees and their interrelationships. The method is based on a combination of 3-D seeded region growing that is used to identify large airways, rule-based two-dimensional (2-D) segmentation of individual CT slices to identify probable locations of smaller diameter airways, and merging of airway regions across the 3-D set of slices resulting in a tree-like airway structure. The method was validated in 40 3-mm-thick CT sections from five data sets of canine lungs scanned via electron beam CT in vivo with lung volume held at a constant pressure. The method's performance was compared with that of the conventional 3-D region growing method. The method substantially outperformed an existing conventional approach to airway tree detection.

The authors have developed a method for fully automated segmentation and labeling of 17 neuroanatomic structures such as thalamus, caudate nucleus, ventricular system, etc. in magnetic resonance (MR) brain images. The authors' method is based on a hypothesize-and-verify principle and uses a genetic algorithm (GA) optimization technique to generate and evaluate image interpretation hypotheses in a feedback loop. The authors' method was trained in 20 individual T1-weighted MR images. Observer-defined contours of neuroanatomic structures were used as a priori knowledge. The method's performance was validated in eight MR images by comparison to observer-defined independent standards. The GA-based image interpretation method correctly interpreted neuroanatomic structures in all images from the test set. Computer-identified and observer-defined neuroanatomic structure areas correlated very well (r=0.99, y=0,95x-2.1). Border positioning errors were small, with a root mean square (rms) border positioning error of 1.5+/-0.6 pixels. The authors' GA-based image interpretation method represents a novel approach to image interpretation and has been shown to produce accurate labeling of neuroanatomic structures in a set of MR brain images.

The authors present a tree pruning strategy used in their novel approach to automated detection of coronary trees in cineangiograms. After the initial steps of the authors' tree detection method, the preliminary tree contains a large number of false branches that must be removed by pruning. The tree pruning approach is based on information about branch absolute and relative lengths, gray level profile information, and information about the structure of the identified tree. The authors have assessed the performance of their tree pruning strategy in 13 coronary angiograms by comparison to manually identified coronary trees. After pruning, the percentage of false branches dropped from 87% to 8%. Computer-detected branch lengths correlated well with observer-identified branch lengths (r=0.95). Pruning substantially improved the accuracy of the detected trees.

Visual estimation of coronary obstruction severity from angiograms suffers from poor inter- and intraobserver reproducibility and is often inaccurate. In spite of the widely recognized limitations of visual analysis, automated methods have not found widespread clinical use, in part because they too frequently fail to accurately identify vessel borders. The authors have developed a robust method for simultaneous detection of left and right coronary borders that is suitable for analysis of complex images with poor contrast, nearby or overlapping structures, or branching vessels. The reliability of the simultaneous border detection method and that of the authors' previously reported conventional border detection method were tested in 130 complex images, selected because conventional automated border detection might be expected to fail. Conventional analysis failed to yield acceptable borders in 65/130 or 50% of images. Simultaneous border detection was much more robust (p<.001) and failed in only 15/130 or 12% of complex images. Simultaneous border detection identified stenosis diameters that correlated significantly better with observer-derived stenosis diameters than did diameters obtained with conventional border detection (p<0.001), Simultaneous detection of left and right coronary borders is highly robust and has substantial promise for enhancing the utility of quantitative coronary angiography in the clinical setting.

Intravascular ultrasound imaging of coronary arteries provides important information about coronary lumen, wall, and plaque characteristics. Quantitative studies of coronary atherosclerosis using intravascular ultrasound and manual identification of wall and plaque borders are limited by the need for observers with substantial experience and the tedious nature of manual border detection. We have developed a method for segmentation of intravascular ultrasound images that identifies the internal and external elastic laminae and the plaque-lumen interface. The border detection algorithm was evaluated in a set of 38 intravascular ultrasound images acquired from fresh cadaveric hearts using a 30 MHz imaging catheter. To assess the performance of our border detection method we compared five quantitative measures of arterial anatomy derived from computer-detected borders with measures derived from borders manually defined by expert observers. Computer-detected and observer-defined lumen areas correlated very well (r=0.96, y=1.02x+0.52), as did plaque areas (r=0.95, y=1.07x-0.48), and percent area stenosis (r=0.93, y=0.99x-1.34.) Computer-derived segmental plaque thickness measurements were highly accurate. Our knowledge-based intravascular ultrasound segmentation method shows substantial promise for the quantitative analysis of in vivo intravascular ultrasound image data.

A method for simultaneous detection of both coronary borders that is based on three-dimensional graph searching principles is presented. The simultaneous method and the authors' previously reported conventional method were applied to 29 coronary images, of which 19 were selected because conventional methods might be expected to have difficulty. Coronary borders identified by the two methods were visually compared. In the 19 difficult images, simultaneous border detection yielded superior results in 7 images and equivalent results in 12 images. Superior or equivalent results were obtained in the remaining 10 typical images. In a set of 43 uncomplicated images, minimal lumen diameters derived using simultaneous border detection correlated well with diameters derived using conventional border detection (r=0.97), diameters obtained from observer-defined borders (r=0.91), and diameters obtained using the Brown-Dodge quantitative coronary arteriography method (r=0.85). Thus simultaneous detection of left and right coronary borders provides improved accuracy in the detection of vessel borders in difficult coronary angiograms.