Abstract Direct volume rendering (DVR) is an important tool for scientific and medical imaging visualization. Modern GPU acceleration has made DVR more accessible; however, the production of high‐quality rendered images with high frame rates is computationally expensive. We propose a deep learning method with a reduced computational demand. We leveraged a conditional generative adversarial network (cGAN) to upsample DVR images (a rendered scene), with a reduced sampling rate to obtain similar visual quality to that of a fully sampled method. Our dvrGAN is combined with a colour‐based loss function that is optimized for DVR images where different structures such as skin, bone, etc. are distinguished by assigning them distinct colours. The loss function highlights the structural differences between images, by examining pixel‐level colour, and thus helps identify, for instance, small bones in the limbs that may not be evident with reduced sampling rates. We evaluated our method in DVR of human computed tomography (CT) and CT angiography (CTA) volumes. Our method retained image quality and reduced computation time when compared to fully sampled methods and outperformed existing state‐of‐the‐art upsampling methods.

Abstract Direct volume rendering (DVR) is a commonly utilized technique for three-dimensional visualization of volumetric medical images. A key goal of DVR is to enable users to visually emphasize regions of interest (ROIs) which may be occluded by other structures. Conventional methods for ROIs visual emphasis require extensive user involvement for the adjustment of rendering parameters to reduce the occlusion, dependent on the user’s viewing direction. Several works have been proposed to automatically preserve the view of the ROIs by eliminating the occluding structures of lower importance in a view-dependent manner. However, they require pre-segmentation labeling and manual importance assignment on the images. An alternative to ROIs segmentation is to use ‘saliency’ to identify important regions. This however lacks semantic information and thus leads to the inclusion of false positive regions. In this study, we propose an attention-driven visual emphasis method for volumetric medical image visualization. We developed a deep learning attention model, termed as focused-class attention map (F-CAM), trained with only image-wise labels for automated ROIs localization and importance estimation. Our F-CAM transfers the semantic information from the classification task for use in the localization of ROIs, with a focus on small ROIs that characterize medical images. Additionally, we propose an attention compositing module that integrates the generated attention map with transfer function within the DVR pipeline to automate the view-dependent visual emphasis of the ROIs. We demonstrate the superiority of our method compared to existing methods on a multi-modality PET-CT dataset and an MRI dataset.

Automatic recognition of 3-D objects in a 3-D model by convolutional neural network (CNN) methods has been successfully applied to various tasks, e.g., robotics and augmented reality. Three-dimensional object recognition is mainly performed by analyzing the object using multi-view images, depth images, graphs, or volumetric data. In some cases, using volumetric data provides the most promising results. However, existing recognition techniques on volumetric data have many drawbacks, such as losing object details on converting points to voxels and the large size of the input volume data that leads to substantial 3-D CNNs. Using point clouds could also provide very promising results; however, point-cloud-based methods typically need sparse data entry and time-consuming training stages. Thus, using volumetric could be a more efficient and flexible recognizer for our special case in the School of Medicine, Shanghai Jiao Tong University. In this article, we propose a novel solution to 3-D object recognition from volumetric data using a combination of three compact CNN models, low-cost SparseNet, and feature representation technique. We achieve an optimized network by estimating extra geometrical information comprising the surface normal and curvature into two separated neural networks. These two models provide supplementary information to each voxel data that consequently improve the results. The primary network model takes advantage of all the predicted features and uses these features in Random Forest (RF) for recognition purposes. Our method outperforms other methods in training speed in our experiments and provides an accurate result as good as the state-of-the-art.

A medical content-based image retrieval (CBIR) system is designed to retrieve images from large imaging repositories that are visually similar to a user′s query image. CBIR is widely used in evidence- based diagnosis, teaching, and research. Although the retrieval accuracy has largely improved, there has been limited development toward visualizing important image features that indicate the similarity of retrieved images. Despite the prevalence of3D volumetric data in medical imaging such as computed tomography (CT), current CBIR systems still rely on 2D cross-sectional views for the visualization of retrieved images. Such 2D visualization requires users to browse through the image stacks to confirm the similarity of the retrieved images and often involves mental reconstruction of 3D information, including the size, shape, and spatial relations of multiple structures. This process is time-consuming and reliant on users’ experience. In this study, we proposed an importance-aware 3D volume visualization method. The rendering parameters were automatically optimized to maximize the visibility of important structures that were detected and prioritized in the retrieval process. We then integrated the proposed visualization into a CBIR system, thereby complementing the 2D cross-sectional views for relevance feedback and further analyses. Our preliminary results demonstrate that 3D visualization can provide additional information using multimodal positron emission tomography and computed tomography (PET- CT) images of a non-small cell lung cancer dataset.

Abstract Angle closure glaucoma leads to fluid deposition in eye, and intraocular pressure occurs that damage the optic nerve, causes blindness and vision loss. Anterior chamber (AC) evaluation is imperative for determining the risk of angle‐closure. Previously, techniques were dependent on either Pentacam–Scheimpflug that interprets poor visual information, anterior segment optical coherence tomography is injurious to intercede opaque optical structures. Therefore, in this paper, an experimental protocol is designed for detailed disease analysis based on IBM SPSS statistics via ultrasound biomicroscopy which is superior in evaluating deep structures; first, the affected parameter for AC is analysed, and afterwards the direction that needs laser surgery is explored. Experiments are conducted on large‐scale clinical studies from an affiliated hospital in Shanghai, China. The dataset comprised 600 AC images in five directions of 60 subjects. The mean with standard deviation for anterior open distance is 0.158790.096779 mm, 0.158630.081435 mm, and anterior chamber angle is 18.74908.0315, 18.74108.3889 for left and right eye respectively. It is found that anterior chamber angle in the downside of the AC is wider than the upside. However, this decision is partly based on the narrowest part of the angle to widen the depth of the direction and eliminate pupil block.

Glaucoma is an ocular disease that causes permanent blindness if not cured at an early stage. Cup-to-disk ratio (CDR), obtained by dividing the height of optic cup (OC) with the height of optic disk (OD), is a widely adopted metric used for glaucoma screening. Therefore, accurately segmenting OD and OC is crucial for calculating a CDR. Most methods have employed deep learning methods for the segmentation of OD and OC. However, these methods are very time consuming. In this article, we present a new fuzzy broad learning system-based technique for OD and OC segmentation with glaucoma screening. We comprehensively integrated extracting a region of interest from RGB images, data augmentation, extracting red and green channel images, and inputting them to the two separate fuzzy broad learning system-based neural networks for segmenting the OD and OC, respectively, and then calculated CDR. Experiments show that our fuzzy broad learning system-based technique outperforms many state-of-the-art methods.

Intracranial blood vessels segmentation from computed tomography angiography (CTA) volumes is a promising biomarker for diagnosis and therapeutic treatment in cerebrovascular diseases. These segmentation outputs are a fundamental requirement in the development of automated decision support systems for preoperative assessment or intraoperative guidance in neuropathology. The state-of-the-art in medical image segmentation methods are reliant on deep learning architectures based on convolutional neural networks. However, despite their popularity, there is a research gap in the current deep learning architectures optimized to address the technical challenges in blood vessel segmentation. These challenges include: (i) the extraction of concrete brain vessels close to the skull; and (ii) the precise marking of the vessel locations. We propose an Optimally Fused Fully end-to-end Network (OFF-eNET) for automatic segmentation of the volumetric 3D intracranial vascular structures. OFF-eNET comprises of three modules. In the first module, we exploit the up-skip connections to enhance information flow, and dilated convolution for detailed preservation of spatial feature map that are designed for thin blood vessels. In the second module, we employ residual mapping along with inception module for speedy network convergence and richer visual representation. For the third module, we make use of the transferred knowledge in the form of cascaded training strategy to gradually optimize the three segmentation stages (basic, complete, and enhanced) to segment thin vessels located close to the skull. All these modules are designed to be computationally efficient. Our OFF-eNET, evaluated using 70 CTA image volumes, resulted in 90.75% performance in the segmentation of intracranial blood vessels and outperformed the state-of-the-art counterparts.

Abstract Direct volume rendering (DVR) visualization helps interpretation because it allows users to focus attention on the subset of volumetric data that is of most interest to them. The ideal visualization of the features of interest (FOIs) in a volume, however, is still a major challenge. The clear depiction of FOIs depends on accurate identification of the FOIs and appropriate specification of the optical parameters via transfer function (TF) design and it is typically a repetitive trial‐and‐error process. We address this challenge by introducing a new method that uses contextual saliency information to group the voxels along a viewing ray into distinct FOIs where ‘contextual saliency’ is a biologically inspired attribute that aids the identification of features that the human visual system considers important. The saliency information is also used to automatically define the optical parameters that emphasize the visual depiction of the FOIs in DVR. We demonstrate the capabilities of our method by its application to a variety of volumetric data sets and highlight its advantages by comparison to current state‐of‐the‐art ray profile analysis methods.

Dual-modality positron emission tomography and computed tomography (PET-CT) depicts pathophysiological function with PET in an anatomical context provided by CT. Three-dimensional volume rendering approaches enable visualization of a two-dimensional slice of interest (SOI) from PET combined with direct volume rendering (DVR) from CT. However, because DVR depicts the whole volume, it may occlude a region of interest, such as a tumor in the SOI. Volume clipping can eliminate this occlusion by cutting away parts of the volume, but it requires intensive user involvement in deciding on the appropriate depth to clip. Transfer functions that are currently available can make the regions of interest visible, but this often requires complex parameter tuning and coupled preprocessing of the data to define the regions. Hence, we propose a new visualization algorithm where an SOI from PET is augmented by volumetric contextual information from a DVR of the counterpart CT so that the obtrusiveness from the CT in the SOI is minimized. Our approach automatically calculates an augmentation depth parameter by considering the occlusion information derived from the voxels of the CT in front of the PET SOI. The depth parameter is then used to generate an opacity weight function that controls the amount of contextual information visible from the DVR. We outline the improvements with our visualization approach compared to other slice-based and our previous approaches. We present the preliminary clinical evaluation of our visualization in a series of PET-CT studies from patients with nonsmall cell lung cancer.

The healthcare industry has begun to utilise web-based systems and cloud computing infrastructure to develop an increasing array of online personal health record (PHR) systems. Although these systems provide the technical capacity to store and retrieve medical data in various multimedia formats, including images, videos, voice, and text, individual patient use remains limited by the lack of intuitive data representation and visualisation techniques. As such, further research is necessary to better visualise and present these records, in ways that make the complex medical data more intuitive. In this study, we present a web-based PHR visualisation system, called the 3D medical graphical avatar (MGA), which was designed to explore web-based delivery of a wide array of medical data types including multi-dimensional medical images; medical videos; text-based data; and spatial annotations. Mapping information was extracted from each of the data types and was used to embed spatial and textual annotations, such as regions of interest (ROIs) and time-based video annotations. Our MGA itself is built from clinical patient imaging studies, when available. We have taken advantage of the emerging web technologies of HTML5 and WebGL to make our application available to a wider base of users and devices. We analysed the performance of our proof-of-concept prototype system on mobile and desktop consumer devices. Our initial experiments indicate that our system can render the medical data in a fashion that enables interactive navigation of the MGA.