OBJECTIVE: This study introduces a novel evaluation framework, GPTRadScore, to systematically assess the performance of multimodal large language models (MLLMs) in generating clinically accurate findings from CT imaging. Specifically, GPTRadScore leverages LLMs as an evaluation metric, aiming to provide a more accurate and clinically informed assessment than traditional language-specific methods. Using this framework, we evaluate the capability of several MLLMs, including GPT-4 with Vision (GPT-4V), Gemini Pro Vision, LLaVA-Med, and RadFM, to interpret findings in CT scans. METHODS: This retrospective study leverages a subset of the public DeepLesion dataset to evaluate the performance of several multimodal LLMs in describing findings in CT slices. GPTRadScore was developed to assess the generated descriptions (location, body part, and type) using GPT-4, alongside traditional metrics. RadFM was fine-tuned using a subset of the DeepLesion dataset with additional labeled examples targeting complex findings. Post fine-tuning, performance was reassessed using GPTRadScore to measure accuracy improvements. RESULTS: Evaluations demonstrated a high correlation of GPTRadScore with clinician assessments, with Pearson's correlation coefficients of 0.87, 0.91, 0.75, 0.90, and 0.89. These results highlight its superiority over traditional metrics, such as BLEU, METEOR, and ROUGE, and indicate that GPTRadScore can serve as a reliable evaluation metric. Using GPTRadScore, it was observed that while GPT-4V and Gemini Pro Vision outperformed other models, significant areas for improvement remain, primarily due to limitations in the datasets used for training. Fine-tuning RadFM resulted in substantial accuracy gains: location accuracy increased from 3.41% to 12.8%, body part accuracy improved from 29.12% to 53%, and type accuracy rose from 9.24% to 30%. These findings reinforce the hypothesis that fine-tuning RadFM can significantly enhance its performance. CONCLUSION: GPT-4 effectively correlates with expert assessments, validating its use as a reliable metric for evaluating multimodal LLMs in radiological diagnostics. Additionally, the results underscore the efficacy of fine-tuning approaches in improving the descriptive accuracy of LLM-generated medical imaging findings.

In chest X-ray (CXR) image analysis, rule-based systems are usually employed to extract labels from reports for dataset releases. However, there is still room for improvement in label quality. These labelers typically output only presence labels, sometimes with binary uncertainty indicators, which limits their usefulness. Supervised deep learning models have also been developed for report labeling but lack adaptability, similar to rule-based systems. In this work, we present MAPLEZ (Medical report Annotations with Privacy-preserving Large language model using Expeditious Zero shot answers), a novel approach leveraging a locally executable Large Language Model (LLM) to extract and enhance findings labels on CXR reports. MAPLEZ extracts not only binary labels indicating the presence or absence of a finding but also the location, severity, and radiologists' uncertainty about the finding. Over eight abnormalities from five test sets, we show that our method can extract these annotations with an increase of 3.6 percentage points (pp) in macro F1 score for categorical presence annotations and more than 20 pp increase in F1 score for the location annotations over competing labelers. Additionally, using the combination of improved annotations and multi-type annotations in classification supervision, we demonstrate substantial advancements in model quality, with an increase of 1.1 pp in AUROC over models trained with annotations from the best alternative approach. We share code and annotations.

Deformable image registration is one of the essential processes in analyzing medical images. In particular, when diagnosing abdominal diseases such as hepatic cancer and lymphoma, multi-domain images scanned from different modalities or different imaging protocols are often used. However, they are not aligned due to scanning times, patient breathing, movement, etc. Although recent learning-based approaches can provide deformations in real-time with high performance, multi-domain abdominal image registration using deep learning is still challenging since the images in different domains have different characteristics such as image contrast and intensity ranges. To address this, this paper proposes a novel unsupervised multi-domain image registration framework using neural optimal transport, dubbed OTMorph. When moving and fixed volumes are given as input, a transport module of our proposed model learns the optimal transport plan to map data distributions from the moving to the fixed volumes and estimates a domain-transported volume. Subsequently, a registration module taking the transported volume can effectively estimate the deformation field, leading to deformation performance improvement. Experimental results on multi-domain image registration using multi-modality and multi-parametric abdominal medical images demonstrate that the proposed method provides superior deformable registration via the domain-transported image that alleviates the domain gap between the input images. Also, we attain the improvement even on out-of-distribution data, which indicates the superior generalizability of our model for the registration of various medical images. Our source code is available at https://github.com/boahK/OTMorph.

Chest radiography, commonly known as CXR, is frequently utilized in clinical settings to detect cardiopulmonary conditions. However, even seasoned radiologists might offer different evaluations regarding the seriousness and uncertainty associated with observed abnormalities. Previous research has attempted to utilize clinical notes to extract abnormal labels for training deep-learning models in CXR image diagnosis. However, these methods often neglected the varying degrees of severity and uncertainty linked to different labels. In our study, we initially assembled a comprehensive new dataset of CXR images based on clinical textual data, which incorporated radiologists' assessments of uncertainty and severity. Using this dataset, we introduced a multi-relationship graph learning framework that leverages spatial and semantic relationships while addressing expert uncertainty through a dedicated loss function. Our research showcases a notable enhancement in CXR image diagnosis and the interpretability of the diagnostic model, surpassing existing state-of-the-art methodologies. The dataset address of disease severity and uncertainty we extracted is: https://physionet.org/content/cad-chest/1.0/.

Multi-parametric magnetic resonance imaging (mpMRI) exams have various series types acquired with different imaging protocols. The DICOM headers of these series often have incorrect information due to the sheer diversity of protocols and occasional technologist errors. To address this, we present a deep learning-based classification model to classify 8 different body mpMRI series types so that radiologists read the exams efficiently. Using mpMRI data from various institutions, multiple deep learning-based classifiers of ResNet, EfficientNet, and DenseNet are trained to classify 8 different MRI series, and their performance is compared. Then, the best-performing classifier is identified, and its classification capability under the setting of different training data quantities is studied. Also, the model is evaluated on the out-of-training-distribution datasets. Moreover, the model is trained using mpMRI exams obtained from different scanners in two training strategies, and its performance is tested. Experimental results show that the DenseNet-121 model achieves the highest F1-score and accuracy of 0.966 and 0.972 over the other classification models with p-value 0.05. The model shows greater than 0.95 accuracy when trained with over 729 studies of the training data, whose performance improves as the training data quantities grow larger. On the external data with the DLDS and CPTAC-UCEC datasets, the model yields 0.872 and 0.810 accuracy for each. These results indicate that in both the internal and external datasets, the DenseNet-121 model attains high accuracy for the task of classifying 8 body MRI series types.

To contribute to automating the medical vision-language model, we propose a novel Chest-Xray Different Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions on both diseases and, more importantly, the differences between them. This is consistent with the radiologist's diagnosis practice that compares the current image with the reference before concluding the report. We collect a new dataset, namely MIMIC-Diff-VQA, including 700,703 QA pairs from 164,324 pairs of main and reference images. Compared to existing medical VQA datasets, our questions are tailored to the Assessment-Diagnosis-Intervention-Evaluation treatment procedure used by clinical professionals. Meanwhile, we also propose a novel expert knowledge-aware graph representation learning model to address this task. The proposed baseline model leverages expert knowledge such as anatomical structure prior, semantic, and spatial knowledge to construct a multi-relationship graph, representing the image differences between two images for the image difference VQA task. The dataset and code can be found at https://github.com/Holipori/MIMIC-Diff-VQA. We believe this work would further push forward the medical vision language model.

Convolutional neural networks (CNNs) have been widely applied to medical images. However, medical images are vulnerable to adversarial attacks by perturbations that are undetectable to human experts. This poses significant security risks and challenges to CNN-based applications in clinic practice. In this work, we quantify the scale of adversarial perturbation imperceptible to clinical practitioners and investigate the cause of the vulnerability in CNNs. Specifically, we discover that noise (i.e., irrelevant or corrupted discriminative information) in medical images might be a key contributor to performance deterioration of CNNs against adversarial perturbations, as noisy features are learned unconsciously by CNNs in feature representations and magnified by adversarial perturbations. In response, we propose a novel defense method by embedding sparsity denoising operators in CNNs for improved robustness. Tested with various state-of-the-art attacking methods on two distinct medical image modalities, we demonstrate that the proposed method can successfully defend against those unnoticeable adversarial attacks by retaining as much as over 90% of its original performance. We believe our findings are critical for improving and deploying CNN-based medical applications in real-world scenarios.

Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in deep learning are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, etc. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.

The latest threat to global health is the COVID-19 outbreak. Although there exist large datasets of chest X-rays (CXR) and computed tomography (CT) scans, few COVID-19 image collections are currently available due to patient privacy. At the same time, there is a rapid growth of COVID-19-relevant articles in the biomedical literature, including those that report findings on radiographs. Here, we present COVID-19-CT-CXR, a public database of COVID-19 CXR and CT images, which are automatically extracted from COVID-19-relevant articles from the PubMed Central Open Access (PMC-OA) Subset. We extracted figures, associated captions, and relevant figure descriptions in the article and separated compound figures into subfigures. Because a large portion of figures in COVID-19 articles are not CXR or CT, we designed a deep-learning model to distinguish them from other figure types and to classify them accordingly. The final database includes 1,327 CT and 263 CXR images (as of May 9, 2020) with their relevant text. To demonstrate the utility of COVID-19-CT-CXR, we conducted four case studies. (1) We show that COVID-19-CT-CXR, when used as additional training data, is able to contribute to improved deep-learning (DL) performance for the classification of COVID-19 and non-COVID-19 CT. (2) We collected CT images of influenza, another common infectious respiratory illness that may present similarly to COVID-19, and fine-tuned a baseline deep neural network to distinguish a diagnosis of COVID-19, influenza, or normal or other types of diseases on CT. (3) We fine-tuned an unsupervised one-class classifier from non-COVID-19 CXR and performed anomaly detection to detect COVID-19 CXR. (4) From text-mined captions and figure descriptions, we compared 15 clinical symptoms and 20 clinical findings of COVID-19 versus those of influenza to demonstrate the disease differences in the scientific publications. Our database is unique, as the figures are retrieved along with relevant text with fine-grained descriptions, and it can be extended easily in the future. We believe that our work is complementary to existing resources and hope that it will contribute to medical image analysis of the COVID-19 pandemic. The dataset, code, and DL models are publicly available at https://github.com/ncbi-nlp/COVID-19-CT-CXR.

Zero-shot learning (ZSL) is one of the most promising avenues of annotation-efficient machine learning. In the era of deep learning, ZSL techniques have achieved unprecedented success. However, the developments of ZSL methods have taken place mostly for natural images. ZSL for medical images has remained largely unexplored. We design a novel strategy for generalized zero-shot diagnosis of chest radiographs. In doing so, we leverage the potential of multi-view semantic embedding, a useful yet less-explored direction for ZSL. Our design also incorporates a self-training phase to tackle the problem of noisy labels alongside improving the performance for classes not seen during training. Through rigorous experiments, we show that our model trained on one dataset can produce consistent performance across test datasets from different sources including those with very different quality. Comparisons with a number of state-of-the-art techniques show the superiority of the proposed method for generalized zero-shot chest x-ray diagnosis.

Prognostic tumor growth modeling via volumetric medical imaging observations can potentially lead to better outcomes of tumor treatment management and surgical planning. Recent advances of convolutional networks (ConvNets) have demonstrated higher accuracy than traditional mathematical models can be achieved in predicting future tumor volumes. This indicates that deep learning based data-driven techniques may have great potentials on addressing such problem. However, current 2D image patch based modeling approaches can not make full use of the spatio-temporal imaging context of the tumor's longitudinal 4D (3D + time) patient data. Moreover, they are incapable to predict clinically-relevant tumor properties, other than the tumor volumes. In this paper, we exploit to formulate the tumor growth process through convolutional Long Short-Term Memory (ConvLSTM) that extract tumor's static imaging appearances and simultaneously capture its temporal dynamic changes within a single network. We extend ConvLSTM into the spatio-temporal domain (ST-ConvLSTM) by jointly learning the inter-slice 3D contexts and the longitudinal or temporal dynamics from multiple patient studies. Our approach can incorporate other non-imaging patient information in an end-to-end trainable manner. Experiments are conducted on the largest 4D longitudinal tumor dataset of 33 patients to date. Results validate that the proposed ST-ConvLSTM model produces a Dice score of 83.2%±5.1% and a RVD of 11.2%±10.8%, both statistically significantly outperforming (p < 0.05) other compared methods of traditional linear model, ConvLSTM, and generative adversarial network (GAN) under the metric of predicting future tumor volumes. Additionally, our new method enables the prediction of both cell density and CT intensity numbers. Last, we demonstrate the generalizability of ST-ConvLSTM by employing it in 4D medical image segmentation task, which achieves an averaged Dice score of 86.3%±1.2% for left-ventricle segmentation in 4D ultrasound with 3 seconds per patient case.

In radiologists' routine work, one major task is to read a medical image, e.g., a CT scan, find significant lesions, and describe them in the radiology report. In this paper, we study the lesion description or annotation problem. Given a lesion image, our aim is to predict a comprehensive set of relevant labels, such as the lesion's body part, type, and attributes, which may assist downstream fine-grained diagnosis. To address this task, we first design a deep learning module to extract relevant semantic labels from the radiology reports associated with the lesion images. With the images and text-mined labels, we propose a lesion annotation network (LesaNet) based on a multilabel convolutional neural network (CNN) to learn all labels holistically. Hierarchical relations and mutually exclusive relations between the labels are leveraged to improve the label prediction accuracy. The relations are utilized in a label expansion strategy and a reliable hard example mining algorithm. We also attach a simple score propagation layer on LesaNet to enhance recall and explore implicit relation between labels. Multilabel metric learning is combined with classification to enable interpretable prediction. We evaluated LesaNet on the public DeepLesion dataset, which contains over 32K diverse lesion images. Experiments show that LesaNet can precisely annotate the lesions using an ontology of 171 fine-grained labels with an average AUC of 0.9344.

Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).

Radiologists in their daily work routinely find and annotate significant abnormalities on a large number of radiology images. Such abnormalities, or lesions, have collected over years and stored in hospitals' picture archiving and communication systems. However, they are basically unsorted and lack semantic annotations like type and location. In this paper, we aim to organize and explore them by learning a deep feature representation for each lesion. A large-scale and comprehensive dataset, DeepLesion, is introduced for this task. DeepLesion contains bounding boxes and size measurements of over 32K lesions. To model their similarity relationship, we leverage multiple supervision information including types, self-supervised location coordinates, and sizes. They require little manual annotation effort but describe useful attributes of the lesions. Then, a triplet network is utilized to learn lesion embeddings with a sequential sampling strategy to depict their hierarchical similarity structure. Experiments show promising qualitative and quantitative results on lesion retrieval, clustering, and classification. The learned embeddings can be further employed to build a lesion graph for various clinically useful applications. An algorithm for intra-patient lesion matching is proposed and validated with experiments.

Tumor growth is associated with cell invasion and mass-effect, which are traditionally formulated by mathematical models, namely reaction-diffusion equations and biomechanics. Such models can be personalized based on clinical measurements to build the predictive models for tumor growth. In this paper, we investigate the possibility of using deep convolutional neural networks to directly represent and learn the cell invasion and mass-effect, and to predict the subsequent involvement regions of a tumor. The invasion network learns the cell invasion from information related to metabolic rate, cell density, and tumor boundary derived from multimodal imaging data. The expansion network models the mass-effect from the growing motion of tumor mass. We also study different architectures that fuse the invasion and expansion networks, in order to exploit the inherent correlations among them. Our network can easily be trained on population data and personalized to a target patient, unlike most previous mathematical modeling methods that fail to incorporate population data. Quantitative experiments on a pancreatic tumor data set show that the proposed method substantially outperforms a state-of-the-art mathematical model-based approach in both accuracy and efficiency, and that the information captured by each of the two subnetworks is complementary.

The chest X-ray is one of the most commonly accessible radiological examinations for screening and diagnosis of many lung diseases. A tremendous number of X-ray imaging studies accompanied by radiological reports are accumulated and stored in many modern hospitals Picture Archiving and Communication Systems (PACS). On the other side, it is still an open question how this type of hospital-size knowledge database containing invaluable imaging informatics (i.e., loosely labeled) can be used to facilitate the data-hungry deep learning paradigms in building truly large-scale high precision computer-aided diagnosis (CAD) systems. In this paper, we present a new chest X-ray database, namely ChestX-ray8, which comprises 108,948 frontal-view X-ray images of 32,717 unique patients with the text-mined eight disease image labels (where each image can have multi-labels), from the associated radiological reports using natural language processing. Importantly, we demonstrate that these commonly occurring thoracic diseases can be detected and even spatially-located via a unified weakly-supervised multi-label image classification and disease localization framework, which is validated using our proposed dataset. Although the initial quantitative results are promising as reported, deep convolutional neural network based reading chest X-rays (i.e., recognizing and locating the common disease patterns trained with only image-level labels) remains a strenuous task for fully-automated high precision CAD systems.

The recent rapid and tremendous success of deep convolutional neural networks (CNN) on many challenging computer vision tasks largely derives from the accessibility of the well-annotated ImageNet and PASCAL VOC datasets. Nevertheless, unsupervised image categorization (i.e., without the ground-truth labeling) is much less investigated, yet critically important and difficult when annotations are extremely hard to obtain in the conventional way of "Google Search" and crowd sourcing. We address this problem by presenting a looped deep pseudo-task optimization (LDPO) framework for joint mining of deep CNN features and image labels. Our method is conceptually simple and rests upon the hypothesized "convergence" of better labels leading to better trained CNN models which in turn feed more discriminative image representations to facilitate more meaningful clusters/labels. Our proposed method is validated in tackling two important applications: 1) Large-scale medical image annotation has always been a prohibitively expensive and easily-biased task even for well-trained radiologists. Significantly better image categorization results are achieved via our proposed approach compared to the previous state-of-the-art method. 2) Unsupervised scene recognition on representative and publicly available datasets with our proposed technique is examined. The LDPO achieves excellent quantitative scene classification results. On the MIT indoor scene dataset, it attains a clustering accuracy of 75:3%, compared to the state-of-the-art supervised classification accuracy of 81:0% (when both are based on the VGG-VD model).

Robust organ segmentation is a prerequisite for computer-aided diagnosis, quantitative imaging analysis, pathology detection, and surgical assistance. For organs with high anatomical variability (e.g., the pancreas), previous segmentation approaches report low accuracies, compared with well-studied organs, such as the liver or heart. We present an automated bottom-up approach for pancreas segmentation in abdominal computed tomography (CT) scans. The method generates a hierarchical cascade of information propagation by classifying image patches at different resolutions and cascading (segments) superpixels. The system contains four steps: 1) decomposition of CT slice images into a set of disjoint boundary-preserving superpixels; 2) computation of pancreas class probability maps via dense patch labeling; 3) superpixel classification by pooling both intensity and probability features to form empirical statistics in cascaded random forest frameworks; and 4) simple connectivity based post-processing. Dense image patch labeling is conducted using two methods: efficient random forest classification on image histogram, location and texture features; and more expensive (but more accurate) deep convolutional neural network classification, on larger image windows (i.e., with more spatial contexts). Over-segmented 2-D CT slices by the simple linear iterative clustering approach are adopted through model/parameter calibration and labeled at the superpixel level for positive (pancreas) or negative (non-pancreas or background) classes. The proposed method is evaluated on a data set of 80 manually segmented CT volumes, using six-fold cross-validation. Its performance equals or surpasses other state-of-the-art methods (evaluated by "leave-one-patient-out"), with a dice coefficient of 70.7% and Jaccard index of 57.9%. In addition, the computational efficiency has improved significantly, requiring a mere 6 ~ 8 min per testing case, versus ≥ 10 h for other methods. The segmentation framework using deep patch labeling confidences is also more numerically stable, as reflected in the smaller performance metric standard deviations. Finally, we implement a multi-atlas label fusion (MALF) approach for pancreas segmentation using the same data set. Under six-fold cross-validation, our bottom-up segmentation method significantly outperforms its MALF counterpart: 70.7±13.0% versus 52.51±20.84% in dice coefficients.

Automation-assisted cervical screening via Pap smear or liquid-based cytology (LBC) is a highly effective cell imaging based cancer detection tool, where cells are partitioned into "abnormal" and "normal" categories. However, the success of most traditional classification methods relies on the presence of accurate cell segmentations. Despite sixty years of research in this field, accurate segmentation remains a challenge in the presence of cell clusters and pathologies. Moreover, previous classification methods are only built upon the extraction of hand-crafted features, such as morphology and texture. This paper addresses these limitations by proposing a method to directly classify cervical cells-without prior segmentation-based on deep features, using convolutional neural networks (ConvNets). First, the ConvNet is pretrained on a natural image dataset. It is subsequently fine-tuned on a cervical cell dataset consisting of adaptively resampled image patches coarsely centered on the nuclei. In the testing phase, aggregation is used to average the prediction scores of a similar set of image patches. The proposed method is evaluated on both Pap smear and LBC datasets. Results show that our method outperforms previous algorithms in classification accuracy (98.3%), area under the curve (0.99) values, and especially specificity (98.3%), when applied to the Herlev benchmark Pap smear dataset and evaluated using five-fold cross validation. Similar superior performances are also achieved on the HEMLBC (H&E stained manual LBC) dataset. Our method is promising for the development of automation-assisted reading systems in primary cervical screening.

Pancreatic neuroendocrine tumors are abnormal growths of hormone-producing cells in the pancreas. Unlike the brain which is protected by the skull, the pancreas can be significantly deformed by its surrounding organs. Consequently, the tumor shape differences observable from images at different time points arise from both tumor growth and pancreatic motion, and tumor growth model personalization may be compromised if such motion is ignored. Therefore, we incorporate pancreatic motion information derived from deformable image registration in model personalization. For more accurate mechanical interactions between tumor growth and pancreatic motion, elastic-growth decomposition is used with a hyperelastic constitutive law to model the mass effect, which allows growth modeling while conserving the mechanical properties. Furthermore, a way of coupling the finite difference method and the finite element method is proposed to greatly reduce the computation time. With both 2-[18F]-fluoro-2-deoxy-D-glucose positron emission tomographic and contrast-enhanced computed tomographic images, functional, structural, and motion data are combined for a patient-specific model. Experiments on synthetic and clinical data show the importance of image-derived motion on estimating pathophysiologically plausible mechanical properties and the promising performance of our framework. From seven patient data sets, the recall, precision, Dice coefficient, relative volume difference, and average surface distance between the personalized tumor growth simulations and the measurements were 83.2 ±8.8%, 86.9 ±8.3%, 84.4 ±4.0%, 13.9 ±9.8%, and 0.6 ±0.1 mm, respectively.

Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normalvs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.

This paper addresses the efficient statistical modeling of multi-organ structures, one of the most challenging scenarios in the medical imaging field due to the frequently limited availability of data. Unlike typical approaches where organs are considered either as single objects or as part of predefined groups, we introduce a more general and natural approach in which all the organs are inter-related inspired by the rhizome theory. Combining canonical correlation analysis with a generalized version of principal component analysis, we propose a new general and flexible framework for multi-organ shape modeling to efficiently characterize the individual organ variability and the relationships between different organs. This new framework called SOMOS can be easily parameterized to mimic a wide variety of alternative statistical shape modeling approaches, including the classic point distribution model, and its more recent multi-resolution variants. The significant superiority of SOMOS over alternative approaches was successfully verified for two different multi-organ databases: six subcortical structures of the brain, and seven abdominal organs. Finally, the organ-prediction capability of the model also significantly outperformed a partial least squared regression-based approach.

In full or arbitrary field-of-view (FOV) 3D CT imaging, obtaining an accurate per-voxel segmentation for complete large and small bones remains an unsolved and challenging problem. The difficulty lies in the notable variation in appearance and position observed among cortical bones, marrow and pathologies. To approach this problem, several studies have employed active shape models and atlas models. In this paper, we argue that a bottom-up approach, defined by classifying and grouping supervoxels, is another viable technique. Moreover, it can be integrated into a conditional random field (CRF) representation. Our approach consists of the following steps: first, an input CT volume is decomposed into supervoxels, in order to ensure very high bone boundary recall. Supervoxels are generated via a robust process of conservative region partitioning and recursive region merging. In order to maximize sparsity and classification efficiency, we use a Bayesian sparse linear classifier to compute and optimize middle-level image features. Next, we disambiguate the CRF unary potentials via contextualized optimization by pooling over selective supervoxel pairs. Finally, we adopt a pairwise support vector machine (SVM) model to learn the CRF pairwise potential in a fully supervised manner. We evaluate our method quantitatively on 137 low-resolution, low-contrast CT volumes with severe imaging noise, among which various bone pathologies are represented. Our system proves to be efficient; it achieves a clinically significant segmentation accuracy level (Dice Coefficient 98.2%).

Despite tremendous progress in computer vision, there has not been an attempt to apply machine learning on very large-scale medical image databases. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's Picture Archiving and Communication System. With natural language processing, we mine a collection of $\sim$216K representative two-dimensional images selected by clinicians for diagnostic reference and match the images with their descriptions in an automated manner. We then employ a weakly supervised approach using all of our available data to build models for generating approximate interpretations of patient images. Finally, we demonstrate a more strictly supervised approach to detect the presence and absence of a number of frequent disease types, providing more specific interpretations of patient scans. A relatively small amount of data is used for this part, due to the challenge in gathering quality labels from large raw text data. Our work shows the feasibility of large-scale learning and prediction in electronic patient records available in most modern clinical institutions. It also demonstrates the trade-offs to consider in designing machine learning systems for analyzing large medical data.

The papers in this special section focus on the technology and applications supported by deep learning. Deep learning is a growing trend in general data analysis and has been termed one of the 10 breakthrough technologies of 2013. Deep learning is an improvement of artificial neural networks, consisting of more layers that permit higher levels of abstraction and improved predictions from data. To date, it is emerging as the leading machine-learning tool in the general imaging and computer vision domains. In particular, convolutional neural networks (CNNs) have proven to be powerful tools for a broad range of computer vision tasks. Deep CNNs automatically learn mid-level and high-level abstractions obtained from raw data (e.g., images). Recent results indicate that the generic descriptors extracted from CNNs are extremely effective in object recognition and localization in natural images. Medical image analysis groups across the world are quickly entering the field and applying CNNs and other deep learning methodologies to a wide variety of applications.

Automated computer-aided detection (CADe) has been an important tool in clinical practice and research. State-of-the-art methods often show high sensitivities at the cost of high false-positives (FP) per patient rates. We design a two-tiered coarse-to-fine cascade framework that first operates a candidate generation system at sensitivities ∼ 100% of but at high FP levels. By leveraging existing CADe systems, coordinates of regions or volumes of interest (ROI or VOI) are generated and function as input for a second tier, which is our focus in this study. In this second stage, we generate 2D (two-dimensional) or 2.5D views via sampling through scale transformations, random translations and rotations. These random views are used to train deep convolutional neural network (ConvNet) classifiers. In testing, the ConvNets assign class (e.g., lesion, pathology) probabilities for a new set of random views that are then averaged to compute a final per-candidate classification probability. This second tier behaves as a highly selective process to reject difficult false positives while preserving high sensitivities. The methods are evaluated on three data sets: 59 patients for sclerotic metastasis detection, 176 patients for lymph node detection, and 1,186 patients for colonic polyp detection. Experimental results show the ability of ConvNets to generalize well to different medical imaging CADe applications and scale elegantly to various data sets. Our proposed methods improve performance markedly in all cases. Sensitivities improved from 57% to 70%, 43% to 77%, and 58% to 75% at 3 FPs per patient for sclerotic metastases, lymph nodes and colonic polyps, respectively.

Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and deep convolutional neural networks (CNNs). CNNs enable learning data-driven, highly representative, hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.

Despite tremendous progress in computer vision, effective learning on very large-scale (> 100K patients) medical image databases has been vastly hindered. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's picture archiving and communication system. Instead of using full 3D medical volumes, we focus on a collection of representative ~216K 2D key images/slices (selected by clinicians for diagnostic reference) with text-driven scalar and vector labels. Our system interleaves between unsupervised learning (e.g., latent Dirichlet allocation, recurrent neural net language models) on document- and sentence-level texts to generate semantic labels and supervised learning via deep convolutional neural networks (CNNs) to map from images to label spaces. Disease-related key words can be predicted for radiology images in a retrieval manner. We have demonstrated promising quantitative and qualitative results. The large-scale datasets of extracted key images and their categorization, embedded vector labels and sentence descriptions can be harnessed to alleviate the deep learning “data-hungry” obstacle in the medical domain.

Due to its importance and possible applications in visualization, tumor detection and preoperative planning, automatic small bowel segmentation is essential for computer-aided diagnosis of small bowel pathology. However, segmenting the small bowel directly on computed tomography (CT) scans is very difficult because of the low image contrast on CT scans and high tortuosity of the small bowel and its close proximity to other abdominal organs. Motivated by the intensity characteristics of abdominal CT images, the anatomic relationship between the mesenteric vasculature and the small bowel, and potential usefulness of the mesenteric vasculature for establishing the path of the small bowel, we propose a novel mesenteric vasculature map-guided method for small bowel segmentation on high-resolution CT angiography scans. The major mesenteric arteries are first segmented using a vessel tracing method based on multi-linear subspace vessel model and Bayesian inference. Second, multi-view, multi-scale vesselness enhancement filters are used to segment small vessels, and vessels directly or indirectly connecting to the superior mesenteric artery are classified as mesenteric vessels. Third, a mesenteric vasculature map is built by linking vessel bifurcation points, and the small bowel is segmented by employing the mesenteric vessel map and fuzzy connectness. The method was evaluated on 11 abdominal CT scans of patients suspected of having carcinoid tumors with manually labeled reference standard. The result, 82.5% volume overlap accuracy compared with the reference standard, shows it is feasible to segment the small bowel on CT scans using the mesenteric vasculature as a roadmap.

Cardiac CT (CCT) is widely available and has been validated for the detection of focal myocardial scar using a delayed enhancement technique in this paper. CCT, however, has not been previously evaluated for quantification of diffuse myocardial fibrosis. In our investigation, we sought to evaluate the potential of low-dose CCT for the measurement of myocardial whole heart extracellular volume (ECV) fraction. ECV is altered under conditions of increased myocardial fibrosis. A framework consisting of three main steps was proposed for CCT whole heart ECV estimation. First, a shape-constrained graph cut (GC) method was proposed for myocardium and blood pool segmentation on postcontrast image. Second, the symmetric demons deformable registration method was applied to register precontrast to postcontrast images. So the correspondences between the voxels from precontrast to postcontrast images were established. Finally, the whole heart ECV value was computed. The proposed method was tested on 20 clinical low-dose CCT datasets with precontrast and postcontrast images. The preliminary results demonstrated the feasibility and efficiency of the proposed method.

In this paper, we propose a new registration method for prone and supine computed tomographic colonography scans using graph matching. We formulate 3-D colon registration as a graph matching problem and propose a new graph matching algorithm based on mean field theory. In the proposed algorithm, we solve the matching problem in an iterative way. In each step, we use mean field theory to find the matched pair of nodes with highest probability. During iterative optimization, one-to-one matching constraints are added to the system in a step-by-step approach. Prominent matching pairs found in previous iterations are used to guide subsequent mean field calculations. The proposed method was found to have the best performance with smallest standard deviation compared with two other baseline algorithms called the normalized distance along the colon centerline (NDACC) ( p = 0.17) with manual colon centerline correction and spectral matching ( p < 1e-5). A major advantage of the proposed method is that it is fully automatic and does not require defining a colon centerline for registration. For the latter NDACC method, user interaction is almost always needed for identifying the colon centerlines.

The paper presents the automated computation of hepatic tumor burden from abdominal computed tomography (CT) images of diseased populations with images with inconsistent enhancement. The automated segmentation of livers is addressed first. A novel 3-D affine invariant shape parameterization is employed to compare local shape across organs. By generating a regular sampling of the organ's surface, this parameterization can be effectively used to compare features of a set of closed 3-D surfaces point-to-point, while avoiding common problems with the parameterization of concave surfaces. From an initial segmentation of the livers, the areas of atypical local shape are determined using training sets. A geodesic active contour corrects locally the segmentations of the livers in abnormal images. Graph cuts segment the hepatic tumors using shape and enhancement constraints. Liver segmentation errors are reduced significantly and all tumors are detected. Finally, support vector machines and feature selection are employed to reduce the number of false tumor detections. The tumor detection true position fraction of 100% is achieved at 2.3 false positives/case and the tumor burden is estimated with 0.9% error. Results from the test data demonstrate the method's robustness to analyze livers from difficult clinical cases to allow the temporal monitoring of patients with hepatic cancer.

In this paper, we present development and testing results for a novel colonic polyp classification method for use as part of a computed tomographic colonography (CTC) computer-aided detection (CAD) system. Inspired by the interpretative methodology of radiologists using 3-D fly-through mode in CTC reading, we have developed an algorithm which utilizes sequences of images (referred to here as videos) for classification of CAD marks. For each CAD mark, we created a video composed of a series of intraluminal, volume-rendered images visualizing the detection from multiple viewpoints. We then framed the video classification question as a multiple-instance learning (MIL) problem. Since a positive (negative) bag may contain negative (positive) instances, which in our case depends on the viewing angles and camera distance to the target, we developed a novel MIL paradigm to accommodate this class of problems. We solved the new MIL problem by maximizing a L2-norm soft margin using semidefinite programming, which can optimize relevant parameters automatically. We tested our method by analyzing a CTC data set obtained from 50 patients from three medical centers. Our proposed method showed significantly better performance compared with several traditional MIL methods.

Traumatic injury of the pelvis is common and potentially devastating, with pelvic fractures being a major cause of trauma patient mortality. Detection and management of pelvic injuries is challenging due to varying injury patterns and resulting complications such as hemorrhage and infection. In this paper, we investigate the application of computer-aided detection (CAD) techniques for pelvic fracture detection. We propose a fast semi-automated method of pelvic fracture detection using a combination of (i) graph cuts and (ii) mean and Gaussian curvatures. A fracture is modeled as a minimum cut in a weighted graph. The same fracture is alternatively modeled as a valley based on the signs of mean and Gaussian curvatures. Each of these methods, in isolation, generates false positives in addition to the true fracture. We then combine the two methods and perform a neighborhood analysis to eliminate the false positives. Experimental results indicate that proposed method is very promising.

The ability to accurately locate a polyp found on computed tomographic colonography (CTC) at subsequent optical colonoscopy (OC) is an important part of CTC for colorectal cancer screening. A previous study has shown that a polyp's normalized distance along the colon centerline (NDACC) derived from CTC data can be utilized to predict its location at OC. We present a novel and automated method using a high degree uniform B-Spline curve fitting of the colon centerline on CTC to simulate OC colonoscope paths in order to more accurately predict polyp position. This evaluation is to determine whether the B-Spline method produces statistically significant improvement over the NDACC method in terms of prediction errors within 5 and 10 cm (± 1 colonoscope mark). Two-tailed Fisher's Exact Tests indicate the B-Spline method is superior to the NDACC method, especially for locating polyps in the central portion of the colon. The ability to predict polyp locations within 5 and 10 cm increased 31.5 % and 10.2 % respectively using the B-Spline method.

Automated detection of blob-like structures is desirable in many biomedical applications such as nodule detection in radiographs and CT images, lymph nodes detection in CT images, and cell counting or tracking in biological images. Multiscale analysis of Hessian matrix is widely used for enhancement or detection of blob-like structures in two-dimensional (2D) and three-dimensional (3D) images. We proposed a new blob detector and a new detection response measure, blobness, based on eigenvalues of the Hessian matrix and local object scale. Pixels with higher blobness are clustered as detected blobs. We evaluated our method by comparison with two existing methods on both simulated and real images. Our results indicated that our automated blob detector had better performance on those images especially when the blobs were close to each other. Our method can be easily extended to 3D for computer-aided detection of blob-like structures in medical images.

In this paper, we propose a new graph matching algorithm based on mean field theory. We first convert the original graph matching problem which is a quadratic integer programming problem to a spin model with quadratic interaction by dropping the matching constraints. Then the matching constraints are added to the system iteratively after each round of mean field calculation. Prominent matching pairs found in previous iterations will guide the mean field calculation in the next round. Experiments on the CMU house dataset and a CTC dataset show promising matching results.

Colon cancer is the second leading cause of cancer-related deaths in the United States. Computed tomographic colonography (CTC) combined with a computer aided detection system provides a feasible approach for improving colonic polyps detection and increasing the use of CTC for colon cancer screening. To distinguish true polyps from false positives, various features extracted from polyp candidates have been proposed. Most of these traditional features try to capture the shape information of polyp candidates or neighborhood knowledge about the surrounding structures (fold, colon wall, etc.). In this paper, we propose a new set of shape descriptors for polyp candidates based on statistical curvature information. These features called histograms of curvature features are rotation, translation and scale invariant and can be treated as complementing existing feature set. Then in order to make full use of the traditional geometric features (defined as group A) and the new statistical features (group B) which are highly heterogeneous, we employed a multiple kernel learning method based on semi-definite programming to learn an optimized classification kernel from the two groups of features. We conducted leave-one-patient-out test on a CTC dataset which contained scans from 66 patients. Experimental results show that a support vector machine (SVM) based on the combined feature set and the semi-definite optimization kernel achieved higher FROC performance compared to SVMs using the two groups of features separately. At a false positive per scan rate of 5, the sensitivity of the SVM using the combined features improved from 0.77 (Group A) and 0.73 (Group B) to 0.83 (p ≤ 0.01).

In this paper, we propose a fast mean-field method called LHMF to handle probabilistic models of large-scale data in high dimensional space. By using diffusion map locally linear embedding method which is a non-linear dimensionality reduction method, we first embed the high dimensional data into a low dimensional space. Then we construct a coarse-grained graph which preserves the spectral properties of original weighted graph in the high dimensional space by clustering. A new spin model is defined in the diffusion space and the geometric centroids of clusters represent variables in the new spin model. The calculation demand of mean-field methods can be reduced greatly on the coarse-grained spin model. The final marginal moments of original variables are derived from the states of geometric centroids by using geometric harmonics. We first tested the proposed method on the MNIST hand-written digits dataset. Experimental results show that the LHMF method is competent with consistency approach, a state-of-the-art semi-supervised learning method. Then we applied the proposed method to a large-scale colonic polyp dataset from computed tomography (CT) scans. Free-response operator characteristic analysis shows that our method achieves higher sensitivity with lower false positive rate compared with support vector machines.

Colon cancer is the second leading cause of cancer-related deaths per year in industrial nations. Virtual colonoscopy is a new, less invasive alternative to the usually practiced optical colonoscopy for colorectal polyp and cancer screening. In this paper, we present some physics-based modeling and pattern recognition techniques to identify anatomical landmarks in the human colon like the haustral folds and the tenia coli to further exploit the benefits of virtual colonoscopy. A combination of heat diffusion field algorithm and fuzzy c-means clustering algorithm is used to detect the haustral folds in human colon from volumetric computed tomography (CT) images. Each voxel on the corresponding colon surface is parameterized using the colon centerline information and associated local Frenet frames. The parameterized fold information is utilized to establish the tentative location of one tenia coli. Preliminary results on automated detection of tenia coli are shown on the colon surface.

Computed tomographic colonography (CTC) provides a feasible way for the detection of colorectal polyps and cancer screening. In the clinical practice of CTC, a true colonic polyp will be confirmed with high confidence if a radiologist can find it in both the supine and prone scans. To assist radiologists in CTC reading, we propose a new colonic polyp matching method based on statistical curvature information of polyp candidates. We first extract histograms of curvature-related features (HCF) from each polyp candidate, then use diffusion map to embed the original high dimensional data into a low-dimensional space. Experimental results show that by using our HCF method, we can improve the sensitivity from 0.58 to 0.74 at false positive rate 0.1 compared with a traditional method that uses only means of curvature-related features.

Ankylosing Spondylitis is a disease characterized by abnormal bone structures (syndesmophytes) growing at intervertebral disk spaces. Because this growth is so slow as to be undetectable on plain radiographs taken over years, it is desirable to resort to computerized techniques to complement qualitative human judgment with precise quantitative measures. We developed an algorithm with minimal user intervention that provides such measures using high-resolution computed tomography (CT) images. To the best of our knowledge it is the first time that determination of the disease's status is attempted by direct measurement of the syndesmophytes. The first part of our algorithm segments the whole vertebral body using a 3-D multiscale cascade of successive level sets. The second part extracts the continuous ridgeline of the vertebral body where syndesmophytes are located. For that we designed a novel level set implementation capable of evolving on the isosurface of an object represented by a triangular mesh using curvature features. The third part of the algorithm segments the syndesmophytes from the vertebral body using local cutting planes and quantitates them. We present experimental work done with 10 patients from each of which we processed five vertebrae. The results of our algorithm were validated by comparison with a semi-quantitative evaluation made by a medical expert who visually inspected the CT scans. Correlation between the two evaluations was found to be 0.936 ( p < 10(-18)) .

CT colonography (CTC) is an emerging noninvasive technique for screening and diagnosing colon cancers. Computer aided detection (CAD) techniques can increase sensitivity and reduce false positives. We propose to employ topographical height maps in our CAD pipeline. For every detection, a height map is computed using a ray-casting algorithm. Since colonic polyps are protrusions outward from the colon wall and are round in contour, their height maps present concentric patterns. The projection direction is optimized through a multi-scale spherical search. We derive several topographic features from the map, and also compute texture features from the Haar wavelet coefficients. We send the selected features to a committee of support vector machines for classification. We have tested our method on 1186 patients with 226 polyps. Results showed that the height map features can reduce false positives by about 50%.

Virtual colonoscopy (VC) is becoming a more prevalent method to detect and diagnose colorectal cancer. An essential component of using VC to detect cancerous polyps, especially in conjunction with computer-aided diagnosis, is the accurate calculation of the centerline of the colon. While the colon is often modeled as a simple cylinder, the amount of colonic distention may vary between patients and within the same patient often causing loops and multiple disconnected segments to be present in the colon segmentation. These variations have caused previous centerline algorithms to fail to capture a complete and accurate centerline for all colons. We have developed an automatic method to determine from a computed tomography (CT) VC a subvoxel precise centerline that is accurate even in cases of over-distended or under-distended colons. In this algorithm, the loops in the colon caused by over-distention are detected and removed when the centerline calculation is performed. Also, a newly developed method for the detection and segmentation of the outer wall of the colon is used to connect collapsed portions of the colon where the lumen segmentation fails to produce a continuous centerline. These two methods allow for a complete and accurate centerline to be calculated in uniformly distended colons as well as in colons containing segments which are over-distended and/or under-distended. We have demonstrated successfully the effectiveness of our algorithm on 50 cases, 25 of which resulted in erroneous solutions by previous centerline algorithms due to variability in the colon distention.

In oral contrast-enhanced CT colonography, patients are given barium- or iodine-containing solutions to drink to tag out remnants of stool and residual fluid in the colon. Frequently, residual fecal matter absorbs more tagging material and appears much brighter on CT images than surrounding opacified fluid. This may cause even advanced segmentation procedures, like fuzzy connectedness, to miss local regions of colonic lumen. This in turn leads to spurious deformations of the reconstructed colonic wall and impairs interpretation. We show that these problems may be avoided when the properly designed asymmetric affinities are used for segmenting air- and fluid-filled parts of the colon. After this improvement, the segmented volume does not contain holes of missed regions and resulting colonic surface is smooth and free from undesired distortion

Reliable segmentation of the colon is a requirement for three-dimensional visualization programs and automatic detection of polyps on computed tomography (CT) colonography. There is an evolving clinical consensus that giving patients positive oral contrast to tag out remnants of stool and residual fluids is mandatory. The presence of positive oral contrast in the colon adds an additional challenge for colonic segmentation but ultimately is beneficial to the patient because the enhanced fluid helps reveal polyps in otherwise hidden areas. Therefore, we developed a new segmentation procedure which can handle both air- and fluid-filled parts of the colon. The procedure organizes individual air- and fluid-filled regions into a graph that enables identification and removal of undesired leakage outside the colon. In addition, the procedure provides a risk assessment of possible leakage to assist the user prior to the tedious task of visual verification. The proposed hybrid algorithm uses modified region growing, fuzzy connectedness and level set segmentation. We tested our algorithm on 160 CT colonography scans containing 183 known polyps. All 183 polyps were in segmented regions. In addition, visual inspection of 24 CT colonography scans demonstrated good performance of our procedure: the reconstructed colonic wall appeared smooth even at the interface between air and fluid and there were no leaked regions.

We present a new method for guiding virtual colonoscopic navigation and registration by using teniae coli as anatomical landmarks. As most existing protocols require a patient to be scanned in both supine and prone positions to increase sensitivity in detecting colonic polyps, reference and registration between scans are necessary. However, the conventional centerline approach, generating only the longitudinal distance along the colon, lacks the necessary orientation information to synchronize the virtual navigation cameras in both scanned positions. In this paper we describe a semi-automatic method to detect teniae coli from a colonic surface model reconstructed from CT colonography. Teniae coli are three bands of longitudinal smooth muscle on the surface of the colon. They form a triple helix structure from the appendix to the sigmoid colon and are ideal references for virtual navigation. Our method was applied to 3 patients resulting in 6 data sets (supine and prone scans). The detected teniae coli matched well with our visual inspection. In addition, we demonstrate that polyps visible on both scans can be located and matched more efficiently with the aid of a teniae coli guided navigation implementation.

Virtual bronchoscopy reconstructions of the airway noninvasively provide useful morphologic information of structural abnormalities such as stenoses and masses. In this paper, we show how virtual bronchoscopy can be used to perform aerodynamic calculations in anatomically realistic models. Pressure and flow patterns in a human airway were computed noninvasively. These showed decreased pressure and increased shear stress in the region of a stenosis.

Virtual endoscopy (VE) is an itegration of medical imaging and virtual reality. VE gives a computer-based alternative to standard radiological image viewing and to traditional fiberoptic endoscopy for examining the interior structures of human organs. It has evolved rapidly in the past decade, as high-resolution helical spiral computed tomography (CT) has evolved for clinical use. A total of 22 submissions were received, covering modalities of virtual colonoscopy, virtual bronchoscopy, and virtual angioscopy, with technical focus on image segmentation, registration, dynamical modeling, flight path planning, fast volumetric rendering, interactive and active navigation, and CAD.

An automatic method to segment colonic polyps in computed tomography (CT) colonography is presented in this paper. The method is based on a combination of knowledge-guided intensity adjustment, fuzzy c-mean clustering, and deformable models. The computer segmentations were compared with manual segmentations to validate the accuracy of our method. An average 76.3% volume overlap percentage among 105 polyp detections was reported in the validation, which was very good considering the small polyp size. Several experiments were performed to investigate the intraoperator and interoperator repeatability of manual colonic polyp segmentation. The investigation demonstrated that the computer-human repeatability was as good as the interoperator repeatability. The polyp segmentation was also applied in computer-aided detection (CAD) to reduce the number of false positive (FP) detections and provide volumetric features for polyp classification. Our segmentation method was able to eliminate 30% of FP detections. The volumetric features computed from the segmentation can further reduce FP detections by 50% at 80% sensitivity.

In the setting of von Hippel-Lindau disease, accurate quantitation of kidney lesions is important for genetic research. Unfortunately, fully automated quantitation is difficult because the lesion boundaries are complex. Therefore, we developed a method to semi-automate the quantitation of these renal lesions. We studied helical CT scans of 10 kidneys from 8 patients with von Hippel-Lindau disease. The kidneys were segmented from surrounding structures using an interactive marker-controlled watershed algorithm. Renal lesions (cysts and solid tumors) were identified using thresholding and then characterized by size using mathematical morphology and granulometry. There were 50 cysts and 16 solid lesions. The mean (/spl plusmn/ sd) numbers of interior and exterior manually placed contours required to perform a complete watershed segmentation of the kidneys were 2.2 /spl plusmn/1.2 and 1.2 /spl plusmn/0.6, respectively. The mean difference between the watershed and manual methods of computing renal volume was 13 /spl plusmn/18 mL (5 /spl plusmn/2% of total renal volume) and is not clinically significant. There was no significant difference between volumes of renal lesions measured manually and using the semi-automated method (p > 0.3).

Confocal volume rendering is a recently described technique to perform segmentation-free rendering. It can be combined with other post-processing techniques to yield images with desirable features such as improved visualization of surface detail and reduced clipping of important anatomy. We show the first application (to our knowledge) of confocal volume rendering to imaging of the thorax. We demonstrate how confocal volume rendering can be used to reveal intrathoracic airways and lung masses.

Interpretation of magnetic resonance angiography (MRA) is problematic due to complexities of vascular shape and to artifacts such as the partial volume effect. We present new methods to assist in the interpretation of MRA. These include methods for detection of vessel paths and for determination of branching patterns of vascular trees. They are based on the ordered region growing (ORG) algorithm that represents the image as an acyclic graph, which can be reduced to a skeleton by specifying vessel endpoints or by a pruning process. Ambiguities in the vessel branching due to vessel overlap are effectively resolved by heuristic methods that incorporate a priori knowledge of bifurcation spacing. Vessel paths are detected at interactive speeds on a 500-MHz processor using vessel endpoints. These methods apply best to smaller vessels where the image intensity peaks at the center of the lumen which, for the abdominal MRA, includes vessels whose diameter is less than 1 cm.