Medical image segmentation is a fundamental task in numerous medical applications. Recently, language-guided segmentation has shown promise in medical scenarios where textual clinical reports are readily available as semantic guidance. Clinical reports contain diagnostic information provided by clinicians, which can provide auxiliary textual semantics to guide segmentation. However, existing language-guided segmentation methods neglect the inherent pattern gaps between image and text modalities, resulting in sub-optimal visual-language integration. Contrastive learning is a well-recognized approach to align image-text patterns, but it has not been optimized for medical image segmentation, where clinically meaningful semantics are often concentrated in localized target regions rather than the entire image. In this study, we propose TMCA, a Target-informed Multi-level Contrastive Alignment framework to bridge image-text pattern gaps for medical language-guided segmentation. The core innovation is to reformulate image-text contrastive alignment from conventional instance-level matching to segmentation-oriented semantic matching, where image-text samples are aligned according to their segmentation targets rather than merely whether they come from the same patient. Specifically, TMCA enables target-informed image-text alignments and fine-grained textual guidance by introducing: (i) a target-sensitive semantic distance module that utilizes target information for more granular image-text alignment modeling, (ii) a multi-level contrastive alignment strategy that directs fine-grained textual guidance to multi-scale image details, and (iii) a language-guided target enhancement module that reinforces attention to critical regions based on the aligned image-text patterns. Extensive experiments on four public benchmarks, involving three medical imaging modalities with clinical reports, show that TMCA enabled superior performance over state-of-the-art language-guided medical segmentation methods.

Abstract Direct volume rendering (DVR) is an important tool for scientific and medical imaging visualization. Modern GPU acceleration has made DVR more accessible; however, the production of high‐quality rendered images with high frame rates is computationally expensive. We propose a deep learning method with a reduced computational demand. We leveraged a conditional generative adversarial network (cGAN) to upsample DVR images (a rendered scene), with a reduced sampling rate to obtain similar visual quality to that of a fully sampled method. Our dvrGAN is combined with a colour‐based loss function that is optimized for DVR images where different structures such as skin, bone, etc. are distinguished by assigning them distinct colours. The loss function highlights the structural differences between images, by examining pixel‐level colour, and thus helps identify, for instance, small bones in the limbs that may not be evident with reduced sampling rates. We evaluated our method in DVR of human computed tomography (CT) and CT angiography (CTA) volumes. Our method retained image quality and reduced computation time when compared to fully sampled methods and outperformed existing state‐of‐the‐art upsampling methods.

Medical image representations can be learned through medical vision-language contrastive learning (mVLCL) where medical imaging reports are used as weak supervision through image-text alignment. These learned image representations can be transferred to and benefit various downstream medical vision tasks such as disease classification and segmentation. Recent mVLCL methods attempt to align image sub-regions and the report keywords as local-matchings. However, these methods aggregate all local-matchings via simple pooling operations while ignoring the inherent relations between them. These methods therefore fail to reason between local-matchings that are semantically related, e.g., local-matchings that correspond to the disease word and the location word (semantic-relations), and also fail to differentiate such clinically important local-matchings from others that correspond to less meaningful words, e.g., conjunction words (importance-relations). Hence, we propose a mVLCL method that models the inter-matching relations between local-matchings via a relation-enhanced contrastive learning framework (RECLF). In RECLF, we introduce a semantic-relation reasoning module (SRM) and an importance-relation reasoning module (IRM) to enable more fine-grained report supervision for image representation learning. We evaluated our method using six public benchmark datasets on four downstream tasks, including segmentation, zero-shot classification, linear classification, and cross-modal retrieval. Our results demonstrated the superiority of our RECLF over the state-of-the-art mVLCL methods with consistent improvements across single-modal and cross-modal tasks. These results suggest that our RECLF, by modeling the inter-matching relations, can learn improved medical image representations with better generalization capabilities.

A medical content-based image retrieval (CBIR) system is designed to retrieve images from large imaging repositories that are visually similar to a user′s query image. CBIR is widely used in evidence- based diagnosis, teaching, and research. Although the retrieval accuracy has largely improved, there has been limited development toward visualizing important image features that indicate the similarity of retrieved images. Despite the prevalence of3D volumetric data in medical imaging such as computed tomography (CT), current CBIR systems still rely on 2D cross-sectional views for the visualization of retrieved images. Such 2D visualization requires users to browse through the image stacks to confirm the similarity of the retrieved images and often involves mental reconstruction of 3D information, including the size, shape, and spatial relations of multiple structures. This process is time-consuming and reliant on users’ experience. In this study, we proposed an importance-aware 3D volume visualization method. The rendering parameters were automatically optimized to maximize the visibility of important structures that were detected and prioritized in the retrieval process. We then integrated the proposed visualization into a CBIR system, thereby complementing the 2D cross-sectional views for relevance feedback and further analyses. Our preliminary results demonstrate that 3D visualization can provide additional information using multimodal positron emission tomography and computed tomography (PET- CT) images of a non-small cell lung cancer dataset.

The identification of melanoma involves an integrated analysis of skin lesion images acquired using clinical and dermoscopy modalities. Dermoscopic images provide a detailed view of the subsurface visual structures that supplement the macroscopic details from clinical images. Visual melanoma diagnosis is commonly based on the 7-point visual category checklist (7PC), which involves identifying specific characteristics of skin lesions. The 7PC contains intrinsic relationships between categories that can aid classification, such as shared features, correlations, and the contributions of categories towards diagnosis. Manual classification is subjective and prone to intra- and interobserver variability. This presents an opportunity for automated methods to aid in diagnostic decision support. Current state-of-the-art methods focus on a single image modality (either clinical or dermoscopy) and ignore information from the other, or do not fully leverage the complementary information from both modalities. Furthermore, there is not a method to exploit the 'intercategory' relationships in the 7PC. In this study, we address these issues by proposing a graph-based intercategory and intermodality network (GIIN) with two modules. A graph-based relational module (GRM) leverages intercategorical relations, intermodal relations, and prioritises the visual structure details from dermoscopy by encoding category representations in a graph network. The category embedding learning module (CELM) captures representations that are specialised for each category and support the GRM. We show that our modules are effective at enhancing classification performance using three public datasets (7PC, ISIC 2017, and ISIC 2018), and that our method outperforms state-of-the-art methods at classifying the 7PC categories and diagnosis.

Multimodal positron emission tomography-computed tomography (PET-CT) is used routinely in the assessment of cancer. PET-CT combines the high sensitivity for tumor detection of PET and anatomical information from CT. Tumor segmentation is a critical element of PET-CT but at present, the performance of existing automated methods for this challenging task is low. Segmentation tends to be done manually by different imaging experts, which is labor-intensive and prone to errors and inconsistency. Previous automated segmentation methods largely focused on fusing information that is extracted separately from the PET and CT modalities, with the underlying assumption that each modality contains complementary information. However, these methods do not fully exploit the high PET tumor sensitivity that can guide the segmentation. We introduce a deep learning-based framework in multimodal PET-CT segmentation with a multimodal spatial attention module (MSAM). The MSAM automatically learns to emphasize regions (spatial areas) related to tumors and suppress normal regions with physiologic high-uptake from the PET input. The resulting spatial attention maps are subsequently employed to target a convolutional neural network (CNN) backbone for segmentation of areas with higher tumor likelihood from the CT image. Our experimental results on two clinical PET-CT datasets of non-small cell lung cancer (NSCLC) and soft tissue sarcoma (STS) validate the effectiveness of our framework in these different cancer types. We show that our MSAM, with a conventional U-Net backbone, surpasses the state-of-the-art lung tumor segmentation approach by a margin of 7.6% in Dice similarity coefficient (DSC).

Fluorodeoxyglucose Positron emission tomography (FDG PET) is the imaging modality of choice for the diagnosis of lung cancer. The automated segmentation of tumors in PET images is a fundamental requirement for image analysis in computer aided diagnosis systems. Current tumor segmentation in PET generally relies on local features to discriminate tumor from the background. These methods are limited due to poor resolution, and subtle inter-class differences when there is normal FDG uptake region (i.e., in the heart and mediastinum) in the same field of view. We propose a new image based discriminative method to separate tumor regions from normal regions. We introduce a convolutional adversarial auto-encoder to learn a latent space which models normal (disease-free) variations of PET images, and then to compute a residual map that identifies where the PET image differs from this manifold due to anomalies, i.e., tumors. Our method is tolerant to normal intra-class variations among the PET images but is discriminative of the tumors with high sensitivity. Our experiments with a clinical lung cancer dataset show that our method outperformed the state-of-the-art unsupervised segmentation methods. We also achieved higher dice score (62.0%) and sensitivity (77.9%) than the supervised U-Net method (59.5% and 59.7%).

The accuracy and robustness of image classification with supervised deep learning are dependent on the availability of large-scale labelled training data. In medical imaging, these large labelled datasets are sparse, mainly related to the complexity in manual annotation. Deep convolutional neural networks (CNNs), with transferable knowledge, have been employed as a solution to limited annotated data through: 1) fine-tuning generic knowledge with a relatively smaller amount of labelled medical imaging data, and 2) learning image representation that is invariant to different domains. These approaches, however, are still reliant on labelled medical image data. Our aim is to use a new hierarchical unsupervised feature extractor to reduce reliance on annotated training data. Our unsupervised approach uses a multi-layer zero-bias convolutional auto-encoder that constrains the transformation of generic features from a pre-trained CNN (for natural images) to non-redundant and locally relevant features for the medical image data. We also propose a context-based feature augmentation scheme to improve the discriminative power of the feature representation. We evaluated our approach on 3 public medical image datasets and compared it to other state-of-the-art supervised CNNs. Our unsupervised approach achieved better accuracy when compared to other conventional unsupervised methods and baseline fine-tuned CNNs.

The analysis of multi-modality positron emission tomography and computed tomography (PET-CT) images for computer aided diagnosis applications (e.g., detection and segmentation) requires combining the sensitivity of PET to detect abnormal regions with anatomical localization from CT. Current methods for PET-CT image analysis either process the modalities separately or fuse information from each modality based on knowledge about the image analysis task. These methods generally do not consider the spatially varying visual characteristics that encode different information across the different modalities, which have different priorities at different locations. For example, a high abnormal PET uptake in the lungs is more meaningful for tumor detection than physiological PET uptake in the heart. Our aim is to improve fusion of the complementary information in multi-modality PET-CT with a new supervised convolutional neural network (CNN) that learns to fuse complementary information for multi-modality medical image analysis. Our CNN first encodes modality-specific features and then uses them to derive a spatially varying fusion map that quantifies the relative importance of each modality's features across different spatial locations. These fusion maps are then multiplied with the modality-specific feature maps to obtain a representation of the complementary multi-modality information at different locations, which can then be used for image analysis. We evaluated the ability of our CNN to detect and segment multiple regions (lungs, mediastinum, tumors) with different fusion requirements using a dataset of PET-CT images of lung cancer. We compared our method to baseline techniques for multi-modality image fusion (fused inputs (FS), multi-branch (MB) techniques, and multichannel (MC) techniques) and segmentation. Our findings show that our CNN had a significantly higher foreground detection accuracy (99.29%, p < 0:05) than the fusion baselines (FS: 99.00%, MB: 99.08%, TC: 98.92%) and a significantly higher Dice score (63.85%) than recent PET-CT tumor segmentation methods.

Automatic event detection in cell videos is essential for monitoring cell populations in biomedicine. Deep learning methods have advantages over traditional approaches for cell event detection due to their ability to capture more discriminative features of cellular processes. Supervised deep learning methods, however, are inherently limited due to the scarcity of annotated data. Unsupervised deep learning methods have shown promise in general (non-cell) videos because they can learn the visual appearance and motion of regularly occurring events. Cell videos, however, can have rapid, irregular changes in cell appearance and motion, such as during cell division and death, which are often the events of most interest. We propose a novel unsupervised two-path input neural network architecture to capture these irregular events with three key elements: 1) a visual encoding path to capture regular spatiotemporal patterns of observed objects with convolutional long short-term memory units; 2) an event detection path to extract information related to irregular events with max-pooling layers; and 3) integration of the hidden states of the two paths to provide a comprehensive representation of the video that is used to simultaneously locate and classify cell events. We evaluated our network in detecting cell division in densely packed stem cells in phase-contrast microscopy videos. Our unsupervised method achieved higher or comparable accuracy to standard and state-of-the-art supervised methods.

The accurate identification of malignant lung nodules on chest CT is critical for the early detection of lung cancer, which also offers patients the best chance of cure. Deep learning methods have recently been successfully introduced to computer vision problems, although substantial challenges remain in the detection of malignant nodules due to the lack of large training data sets. In this paper, we propose a multi-view knowledge-based collaborative (MV-KBC) deep model to separate malignant from benign nodules using limited chest CT data. Our model learns 3-D lung nodule characteristics by decomposing a 3-D nodule into nine fixed views. For each view, we construct a knowledge-based collaborative (KBC) submodel, where three types of image patches are designed to fine-tune three pre-trained ResNet-50 networks that characterize the nodules' overall appearance, voxel, and shape heterogeneity, respectively. We jointly use the nine KBC submodels to classify lung nodules with an adaptive weighting scheme learned during the error back propagation, which enables the MV-KBC model to be trained in an end-to-end manner. The penalty loss function is used for better reduction of the false negative rate with a minimal effect on the overall performance of the MV-KBC model. We tested our method on the benchmark LIDC-IDRI data set and compared it to the five state-of-the-art classification approaches. Our results show that the MV-KBC model achieved an accuracy of 91.60% for lung nodule classification with an AUC of 95.70%. These results are markedly superior to the state-of-the-art approaches.

Abstract Direct volume rendering (DVR) visualization helps interpretation because it allows users to focus attention on the subset of volumetric data that is of most interest to them. The ideal visualization of the features of interest (FOIs) in a volume, however, is still a major challenge. The clear depiction of FOIs depends on accurate identification of the FOIs and appropriate specification of the optical parameters via transfer function (TF) design and it is typically a repetitive trial‐and‐error process. We address this challenge by introducing a new method that uses contextual saliency information to group the voxels along a viewing ray into distinct FOIs where ‘contextual saliency’ is a biologically inspired attribute that aids the identification of features that the human visual system considers important. The saliency information is also used to automatically define the optical parameters that emphasize the visual depiction of the FOIs in DVR. We demonstrate the capabilities of our method by its application to a variety of volumetric data sets and highlight its advantages by comparison to current state‐of‐the‐art ray profile analysis methods.

The classification of medical images and illustrations from the biomedical literature is important for automated literature review, retrieval, and mining. Although deep learning is effective for large-scale image classification, it may not be the optimal choice for this task as there is only a small training dataset. We propose a combined deep and handcrafted visual feature (CDHVF) based algorithm that uses features learned by three fine-tuned and pretrained deep convolutional neural networks (DCNNs) and two handcrafted descriptors in a joint approach. We evaluated the CDHVF algorithm on the ImageCLEF 2016 Subfigure Classification dataset and it achieved an accuracy of 85.47%, which is higher than the best performance of other purely visual approaches listed in the challenge leaderboard. Our results indicate that handcrafted features complement the image representation learned by DCNNs on small training datasets and improve accuracy in certain medical image classification problems.

The segmentation of skin lesions in dermoscopic images is a fundamental step in automated computer-aided diagnosis of melanoma. Conventional segmentation methods, however, have difficulties when the lesion borders are indistinct and when contrast between the lesion and the surrounding skin is low. They also perform poorly when there is a heterogeneous background or a lesion that touches the image boundaries; this then results in under- and oversegmentation of the skin lesion. We suggest that saliency detection using the reconstruction errors derived from a sparse representation model coupled with a novel background detection can more accurately discriminate the lesion from surrounding regions. We further propose a Bayesian framework that better delineates the shape and boundaries of the lesion. We also evaluated our approach on two public datasets comprising 1100 dermoscopic images and compared it to other conventional and state-of-the-art unsupervised (i.e., no training required) lesion segmentation methods, as well as the state-of-the-art unsupervised saliency detection methods. Our results show that our approach is more accurate and robust in segmenting lesions compared to other methods. We also discuss the general extension of our framework as a saliency optimization algorithm for lesion segmentation.

Dual-modality positron emission tomography and computed tomography (PET-CT) depicts pathophysiological function with PET in an anatomical context provided by CT. Three-dimensional volume rendering approaches enable visualization of a two-dimensional slice of interest (SOI) from PET combined with direct volume rendering (DVR) from CT. However, because DVR depicts the whole volume, it may occlude a region of interest, such as a tumor in the SOI. Volume clipping can eliminate this occlusion by cutting away parts of the volume, but it requires intensive user involvement in deciding on the appropriate depth to clip. Transfer functions that are currently available can make the regions of interest visible, but this often requires complex parameter tuning and coupled preprocessing of the data to define the regions. Hence, we propose a new visualization algorithm where an SOI from PET is augmented by volumetric contextual information from a DVR of the counterpart CT so that the obtrusiveness from the CT in the SOI is minimized. Our approach automatically calculates an augmentation depth parameter by considering the occlusion information derived from the voxels of the CT in front of the PET SOI. The depth parameter is then used to generate an opacity weight function that controls the amount of contextual information visible from the DVR. We outline the improvements with our visualization approach compared to other slice-based and our previous approaches. We present the preliminary clinical evaluation of our visualization in a series of PET-CT studies from patients with nonsmall cell lung cancer.

The availability of medical imaging data from clinical archives, research literature, and clinical manuals, coupled with recent advances in computer vision offer the opportunity for image-based diagnosis, teaching, and biomedical research. However, the content and semantics of an image can vary depending on its modality and as such the identification of image modality is an important preliminary step. The key challenge for automatically classifying the modality of a medical image is due to the visual characteristics of different modalities: some are visually distinct while others may have only subtle differences. This challenge is compounded by variations in the appearance of images based on the diseases depicted and a lack of sufficient training data for some modalities. In this paper, we introduce a new method for classifying medical images that uses an ensemble of different convolutional neural network (CNN) architectures. CNNs are a state-of-the-art image classification technique that learns the optimal image features for a given classification task. We hypothesise that different CNN architectures learn different levels of semantic image representation and thus an ensemble of CNNs will enable higher quality features to be extracted. Our method develops a new feature extractor by fine-tuning CNNs that have been initialized on a large dataset of natural images. The fine-tuning process leverages the generic image features from natural images that are fundamental for all images and optimizes them for the variety of medical imaging modalities. These features are used to train numerous multiclass classifiers whose posterior probabilities are fused to predict the modalities of unseen images. Our experiments on the ImageCLEF 2016 medical image public dataset (30 modalities; 6776 training images, and 4166 test images) show that our ensemble of fine-tuned CNNs achieves a higher accuracy than established CNNs. Our ensemble also achieves a higher accuracy than methods in the literature evaluated on the same benchmark dataset and is only overtaken by those methods that source additional training data.

Content-based image retrieval (CBIR) is a search technique based on the similarity of visual features and has demonstrated potential benefits for medical diagnosis, education, and research. However, clinical adoption of CBIR is partially hindered by the difference between the computed image similarity and the user's search intent, the semantic gap, with the end result that relevant images with outlier features may not be retrieved. Furthermore, most CBIR algorithms do not provide intuitive explanations as to why the retrieved images were considered similar to the query (e.g., which subset of features were similar), hence, it is difficult for users to verify if relevant images, with a small subset of outlier features, were missed. Users, therefore, resort to examining irrelevant images and there are limited opportunities to discover these "missed" images. In this paper, we propose a new approach to medical CBIR by enabling a guided visual exploration of the search space through a tool, called visual analytics for medical image retrieval (VAMIR). The visual analytics approach facilitates interactive exploration of the entire dataset using the query image as a point-of-reference. We conducted a user study and several case studies to demonstrate the capabilities of VAMIR in the retrieval of computed tomography images and multimodality positron emission tomography and computed tomography images.

Medical images usually exhibit large intra-class variation and inter-class ambiguity in the feature space, which could affect classification accuracy. To tackle this issue, we propose a new Large Margin Local Estimate (LMLE) classification model with sub-categorization based sparse representation. We first sub-categorize the reference sets of different classes into multiple clusters, to reduce feature variation within each subcategory compared to the entire reference set. Local estimates are generated for the test image using sparse representation with reference subcategories as the dictionaries. The similarity between the test image and each class is then computed by fusing the distances with the local estimates in a learning-based large margin aggregation construct to alleviate the problem of inter-class ambiguity. The derived similarities are finally used to determine the class label. We demonstrate that our LMLE model is generally applicable to different imaging modalities, and applied it to three tasks: interstitial lung disease (ILD) classification on high-resolution computed tomography (HRCT) images, phenotype binary classification and continuous regression on brain magnetic resonance (MR) imaging. Our experimental results show statistically significant performance improvements over existing popular classifiers.

Accurate liver segmentation from computed tomography (CT) images is problematic due to non-uniform density, weak boundaries and because there may be multiple liver tumors that have heterogeneous intensities in region(s) of interest (ROIs). So we propose a generalized energy framework that harnesses the statistical intensity approximation with image data on graphs. Our statistical energy term takes advantage of the mixture-of-mixtures Gaussian model to approximate the probability density distribution of the liver and background to better differentiate between the two. The probability density estimation can be combined with the spatial cohesion of the graph-based Dirichlet integral by using graph calculus. Matrix decomposition and differentiation are used to minimize our proposed energy functional. We tested our approach on 20 public high-contrast CT images with single and multiple liver tumors. Our method had an average dice similarity coefficient (DSC) of 93.75±1.29%, an average false positive (FP) rate of 9.43±3.52% and an average false negative (FN) rate of 3.48±1.48%. Our method outperformed the benchmark graph-based Random Walker algorithm (average DSC=81.97±4.09%, average FP rate 34.10±10.53%, and average FN rate 7.10±4.35%).

We present a lesion detection and characterization method for (18)F-fluorodeoxyglucose positron emission tomography-computed tomography (FDG PET-CT) images of the thorax in the evaluation of patients with primary nonsmall cell lung cancer (NSCLC) with regional nodal disease. Lesion detection can be difficult due to low contrast between lesions and normal anatomical structures. Lesion characterization is also challenging due to similar spatial characteristics between the lung tumors and abnormal lymph nodes. To tackle these problems, we propose a context driven approximation (CDA) method. There are two main components of our method. First, a sparse representation technique with region-level contexts was designed for lesion detection. To discriminate low-contrast data with sparse representation, we propose a reference consistency constraint and a spatial consistent constraint. Second, a multi-atlas technique with image-level contexts was designed to represent the spatial characteristics for lesion characterization. To accommodate inter-subject variation in a multi-atlas model, we propose an appearance constraint and a similarity constraint. The CDA method is effective with a simple feature set, and does not require parametric modeling of feature space separation. The experiments on a clinical FDG PET-CT dataset show promising performance improvement over the state-of-the-art.

Graph-based content-based image retrieval (CBIR) techniques, which use graphs to represent image features and calculate image similarity using the graph edit distance, achieve high retrieval accuracy. However, such techniques suffer from high computational complexity. In this paper, we present a graph-based CBIR algorithm that achieves improved retrieval efficiency. We compute a vector space embedding for every graph, using their distances from a set of prototype graphs, so that each vector component represents a distortion from a prototype. This process is performed offline. We compare images by computing the Euclidean distance of the vector embeddings, which is a faster process than calculating the graph edit distance. We evaluated our work using 50 combined positron emission tomography and computed tomography (PET-CT) volumes of patients with lung tumours. Our results show that our method is at least 21 times faster than the graph edit distance with a mean average precision difference of less than 4%.

The healthcare industry has begun to utilise web-based systems and cloud computing infrastructure to develop an increasing array of online personal health record (PHR) systems. Although these systems provide the technical capacity to store and retrieve medical data in various multimedia formats, including images, videos, voice, and text, individual patient use remains limited by the lack of intuitive data representation and visualisation techniques. As such, further research is necessary to better visualise and present these records, in ways that make the complex medical data more intuitive. In this study, we present a web-based PHR visualisation system, called the 3D medical graphical avatar (MGA), which was designed to explore web-based delivery of a wide array of medical data types including multi-dimensional medical images; medical videos; text-based data; and spatial annotations. Mapping information was extracted from each of the data types and was used to embed spatial and textual annotations, such as regions of interest (ROIs) and time-based video annotations. Our MGA itself is built from clinical patient imaging studies, when available. We have taken advantage of the emerging web technologies of HTML5 and WebGL to make our application available to a wider base of users and devices. We analysed the performance of our proof-of-concept prototype system on mobile and desktop consumer devices. Our initial experiments indicate that our system can render the medical data in a fashion that enables interactive navigation of the MGA.

Intensity inhomogeneities and different types/levels of image noise are the two major obstacles to accurate image segmentation by region-based level set models. To provide a more general solution to these challenges, we propose a novel segmentation model that considers global and local image statistics to eliminate the influence of image noise and to compensate for intensity inhomogeneities. In our model, the global energy derived from a Gaussian model estimates the intensity distribution of the target object and background; the local energy derived from the mutual influences of neighboring pixels can eliminate the impact of image noise and intensity inhomogeneities. The robustness of our method is validated on segmenting synthetic images with/without intensity inhomogeneities, and with different types/levels of noise, including Gaussian noise, speckle noise, and salt and pepper noise, as well as images from different medical imaging modalities. Quantitative experimental comparisons demonstrate that our method is more robust and more accurate in segmenting the images with intensity inhomogeneities than the local binary fitting technique and its more recent systematic model. Our technique also outperformed the region-based Chan–Vese model when dealing with images without intensity inhomogeneities and produce better segmentation results than the graph-based algorithms including graph-cuts and random walker when segmenting noisy images.

Segmentation of the target object(s) from images that have multiple complicated regions, mixture intensity distributions or are corrupted by noise poses a challenge for the level set models. In addition, the conventional piecewise smooth level set models normally require prior knowledge about the number of image segments. To address these problems, we propose a novel segmentation energy function with two distribution descriptors to model the background and the target. The single background descriptor models the heterogeneous background with multiple regions. Then, the target descriptor takes into account the intensity distribution and incorporates local spatial constraint. Our descriptors, which have more complete distribution information, construct the unique energy function to differentiate the target from the background and are more tolerant of image noise. We compare our approach to three other level set models: 1) the Chan-Vese; 2) the multiphase level set; and 3) the geodesic level set. This comparison using 260 synthetic images with varying levels and types of image noise and medical images with more complicated backgrounds showed that our method outperforms these models for accuracy and immunity to noise. On an additional set of 300 synthetic images, our model is also less sensitive to the contour initialization as well as to different types and levels of noise.

Equation (16) in the above paper (ibid., vol. 22, no. 8, pp. 3296-3309, Aug. 2013) contained an error in the numerator. Equation (17) in the same paper contained errors in both the numerator and the denominator. The corrected versions of both equations are presented here.

We propose a novel joint probabilistic model that correlates a new probabilistic shape model with the corresponding global intensity distribution to segment multiple abdominal organs simultaneously. Our probabilistic shape model estimates the probability of an individual voxel belonging to the estimated shape of the object. The probability density of the estimated shape is derived from a combination of the shape variations of target class and the observed shape information. To better capture the shape variations, we used probabilistic principle component analysis optimized by expectation maximization to capture the shape variations and reduce computational complexity. The maximum a posteriori estimation was optimized by the iterated conditional mode-expectation maximization. We used 72 training datasets including low- and high-contrast CT images to construct the shape models for the liver, spleen and both kidneys. We evaluated our algorithm on 40 test datasets that were grouped into normal (34 normal cases) and pathologic (6 datasets) classes. The testing datasets were from different databases and manual segmentation was performed by different clinicians. We measured the volumetric overlap percentage error, relative volume difference, average square symmetric surface distance, false positive rate and false negative rate and our method achieved accurate and robust segmentation for multiple abdominal organs simultaneously.

Content-based image retrieval (CBIR) is an image search technique that utilises visual features as search criteria; it has potential clinical applications in evidence-based diagnosis, physician training, and biomedical research. Graph-based CBIR techniques have high accuracy when retrieving images by the similarity of the spatial arrangement of their constituent objects but these techniques were initially designed for single-modality images and have limited retrieval capabilities when multi-modality images, such as combined positron emission tomography and computed tomography (PET-CT), are considered. In this paper, we present a graph-based CBIR approach for multimodality images that integrates modality-specific features on graph vertices and adapts a well-established graph similarity scheme to account for varying vertex feature sets. Furthermore, we propose a graph pruning method that removes redundant edges using the spatial proximity of image regions. We evaluated our work using two simulated data sets, consisting of 2D liver shapes and 3D whole-body lymphoma images. In our experiments we achieved a higher level of retrieval precision using our graph method when compared to conventional graph-based retrieval, demonstrating that our proposed method enabled new capabilities and improved multi-modality CBIR.

Combined PET-CT is now increasingly used for the clinical evaluation of cancer and is arguably the best tool to stage non-small cell lung cancer (NSCLC). We propose a framework to better delineate lung tumors which utilizes information from PET and CT images. The framework is based on a downhill region growing technique for PET and a Gaussian mixture model for CT images. We applied our framework in 20 PET-CT studies from patients with NSCLC. Experiments show that our method is able to delineate lung tumors in complex cases where the tumors are located near other organs with similar intensities in PET images or when the tumors extends into the chest wall or the mediastinum. We also compared 10 of the datasets with experts performing manual delineation, which produced a volumetric overlapped fraction of 0.78 ± 0.10.

Reliable automated or semiautomated lung tumor delineation methods in positron emission tomography should provide accurate tumor boundary definition and separation of the lung tumor from surrounding tissue or "hot spots" that have similar intensities to the lung tumor. We propose a tumor-customized downhill (TCD) method to achieve these objectives. Our approach includes: 1) automatic formulation of a tumor-customized criterion to improve tumor boundary definition, 2) a monotonic property of the standardized uptake value (SUV) of tumors to separate the tumor from adjacent regions of increased metabolism ("hot spot"), and 3) accounts for tumor heterogeneity. Three simulated lesions and 30 PET-CT studies, grouped into "simple" and "complex" groups, were used for evaluation. Our main findings are that TCD, when compared to the threshold based on 40% and 50% maximum SUV, adaptive threshold, Fuzzy c-means, and watershed techniques achieved the highest Dice's similarity coefficient average for simulation data (0.73) and "complex" group (0.71); the least volumetric error in the "simple" (1.76 mL) and the "complex" group (14.59 mL); and TCD solves the problem of leakage into adjacent tissues when many other techniques fail.

The volume and complexity of neurological images have significantly increased, which leads to challenges in efficient data management and retrieval. In this paper, we developed a new content-based image retrieval framework with the localized multiscale Discrete Curvelet Transform (DCvT) features extracted from parametric neurological images. We also compared the performance of three different irregular-to-regular shape padding methods. 142 patient data with neurodegenerative disorders were used in the evaluation. The preliminary results show that our proposed framework supports fast neuroimaging retrieval, and the orthographic projection method can reduce the computational complexity and has a great potential to improve the retrieval for indefinite cases.

This paper presents a framework for effective and fast content-based image retrieval for multi-modality PET-CT lung scans. PET-CT scans present significant advantages in tumor staging, but also place new challenges in computerized image analysis and retrieval. Our framework comprises 5 major components: lung field estimation, texture feature extraction, feature categorization, refinement using SVM, and similarity measure. Clinical data from lung cancer patients are used as case studies, and effective retrieval performance is demonstrated.

Functional neuroimaging has an important role in non-invasive diagnosis of neurodegenerative disorders. There are now large volumes of imaging data generated by functional imaging technologies and so there is a need to efficiently manage and retrieve these data. In this paper, we propose a new scheme for efficient 3D content-based neurological image retrieval. 3D pathology-centric masks were adaptively designed and applied for extracting CMRGlc (cerebral metabolic rate of glucose consumption) texture features with volumetric co-occurrence matrices from neurological FDG PET images. Our results, using 93 clinical dementia studies, show that our approach offers a robust and efficient retrieval mechanism for relevant clinical cases and provides advantages in image data analysis and management.

Automated liver segmentation is problematic due to variations in liver shape / size and because the liver has a similar density distribution to surrounding structures. We propose a method that: 1) utilizes iteratively constructed probabilistic liver and rib cage atlases, 2) conducts the Gaussian distribution analysis to avoid incorrectly classifying the irrelevant surrounding tissues as `liver region' in the conventional probabilistic atlas based method, and maps the intensity range of the input candidate liver region onto the liver atlas, 3) retrieves the `missing parts' of the liver by deformable registration. Our approach is automated and able to segment the liver from high-contrast and low-contrast CT volumes. Forty clinical CT studies were used for atlas construction and validation. Our method outperformed two other probabilistic atlas-based liver segmentation methods.

Functional imaging can provide quantitative functional parameters to aid early diagnosis. Low signal to noise ratio (SNR) in functional imaging, especially for single photon emission computed tomography, poses a challenge in generating voxel-wise parametric images due to unreliable or physiologically meaningless parameter estimates. Our aim was to systematically investigate the performance of our recently proposed adaptive fuzzy clustering (AFC) technique, which applies standard fuzzy clustering to sub-divided data. Monte Carlo simulations were performed to generate noisy dynamic SPECT data with quantitative analysis for the fitting using the general linear least square method (GLLS) and enhanced model-aided GLLS methods. The results show that AFC substantially improves computational efficiency and obtains improved reliability as standard fuzzy clustering in estimating parametric images but is prone to slight underestimation. Normalization of tissue time activity curves may lead to severe overestimation for small structures when AFC is applied.

Dual modality PET/CT has now essentially replaced PET in clinical practice and provided an opportunity to improve image segmentation through the high resolution, lower noise CT data. Thus far most research efforts have concentrated on segmentation of PET-only data. In this work we propose a systematic solution for the automated segmentation of brain PET/CT images into gray, white matter and CSF regions with the MAP-MRF model. Our approach takes advantage of the full information available from the combined scan. A PET/CT image pair and its segmentation result are modelled as a random field triplet, and segmentation is eventually achieved by solving a maximum a posteriori (MAP) problem using the expectation-maximization (EM) algorithm with simulated annealing. We compared the novel algorithm to two widely used PET-only based segmentation methods in the SPM5 toolbox and the VBM toolbox for simulation and patient data. Our results suggest that using the proposed approach substantially improves the accuracy of the delineation of brain structures.

Dynamic imaging with positron emission tomography (PET) is widely used for the in vivo measurement of regional cerebral metabolic rate for glucose (rCMRGlc) with [18F]fluorodeoxy-D-glucose (FDG) and is used for the clinical evaluation of neurological disease. However, in addition to the acquisition of dynamic images, continuous arterial blood sampling is the conventional method to obtain the tracer time-activity curve in blood (or plasma) for the numeric estimation of rCMRGlc in mg glucose/100-g tissue/min. The insertion of arterial lines and the subsequent collection and processing of multiple blood samples are impractical for clinical PET studies because it is invasive, has the remote, but real potential for producing limb ischemia, and it exposes personnel to additional radiation and risks associated with handling blood. In this paper, based on our previously proposed method for extracting kinetic parameters from dynamic PET images, we developed a modified version (post-estimation method) to improve the numerical identifiability of the parameter estimates when we deal with data obtained from clinical studies. We applied both methods to dynamic neurologic FDG PET studies in three adults. We found that the input function and parameter estimates obtained with our noninvasive methods agreed well with those estimated from the gold standard method of arterial blood sampling and that rCMRGlc estimates were highly correlated (r = 0.973). More importantly, no significant difference was found between rCMRGlc estimated by our methods and the gold standard method (P > 0.16). We suggest that our proposed noninvasive methods may offer an advance over existing methods.