We consider the problem of abnormality localization for clinical applications. While deep learning has driven much recent progress in medical imaging, many clinical challenges are not fully addressed, limiting its broader usage. While recent methods report high diagnostic accuracies, physicians have concerns trusting these algorithm results for diagnostic decision-making purposes because of a general lack of algorithm decision reasoning and interpretability. One potential way to address this problem is to further train these models to localize abnormalities in addition to just classifying them. However, doing this accurately will require a large amount of disease localization annotations by clinical experts, a task that is prohibitively expensive to accomplish for most applications. In this work, we take a step towards addressing these issues by means of a new attention-driven weakly supervised algorithm comprising a hierarchical attention mining framework that unifies activation- and gradient-based visual attention in a holistic manner. Our key algorithmic innovations include the design of explicit ordinal attention constraints, enabling principled model training in a weakly-supervised fashion, while also facilitating the generation of visual-attention-driven model explanations by means of localization cues. On two large-scale chest X-ray datasets (NIH ChestX-ray14 and CheXpert), we demonstrate significant localization performance improvements over the current state of the art while also achieving competitive classification performance.

Lung cancer is the leading cause of cancer-related deaths worldwide, while the risk factors for lung cancer mortality can be significantly reduced if the accurate early diagnoses for small malignant lung nodules are possible. In this paper, we propose a hierarchical attention-based multiple instance learning (HA-MIL) framework for patient-level lung cancer diagnosis by introducing two-level cascaded attention mechanisms, one at nodule level and the other at attribute level. The proposed HA-MIL framework is constructed by aggregating important attribute representation into nodule representation and then aggregating important nodule representation into lung cancer representation. The experiments on the public Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset showed that the HA-MIL model performed significantly better than the previous approaches such as the higher-order transfer learning, instance-space MIL and embedding-space MIL, which demonstrated the effectiveness of hierarchical multiple instance learning based on two-level attentions. The results analysis suggested that the HA-MIL model also found the key nodules and attributes by higher attention weights, which were more interpretable for the model decision making.

Automatic pneumothorax segmentation on chest X-ray images is very crucial for diagnosis and treatment as large pneumothorax could be fatal. The pneumothorax segmentation is challenging, as some small pneumothoraces can be subtle, and may overlap with the ribs and clavicles. Meanwhile, the shape variation of pneumothorax is also very large, which also makes the segmentation more difficult. In this paper, we propose a novel automated pneumothorax segmentation framework which consists of three modules: 1) a fully convolutional DenseNet (FC-DenseNet), 2) a spatial and channel squeeze and excitation module (scSE), and 3) a multi-scale module. In order to improve boundary segmentation accuracy, a novel spatial weighted cross-entropy loss function is proposed, which penalize the target, background and contour pixels with different weights. Extensive experiments are conducted on the 2213 chest X-ray images of testing data and the results suggest that proposed segmentation algorithm outperforms the state-of-the-art methods in terms of mean pixel-wise accuracy (MPA) of 0.93±0.13 and dice similarity coefficient (DSC) of 0.92±0.14 etc. Accordingly, the effectiveness of our method is corroborated.

Attributes like texture, lobulation, malignancy, etc., are commonly used to describe the phenotype of a pulmonary nodule in computed tomography (CT) image, which can provide useful medical knowledge for the identification of early stage lung cancer. There may exist certain relations among these attributes, and some attributes may naturally imply or boost others that have been less comprehensively exploited in previous studies. In this paper, we explicitly model the relations among 11 attributes of nodules by way of transfer learning and extract a meta-structure that captures the transferabilities across deep features of these attributes. Specifically, a higher-order transfer learning scheme is proposed by involving three phases, i.e., semantic attribute-specific modeling, semantic attributes transfer modeling and pathologic attribute generalizing, to explore the strongest association across various attributes and to boost the nodule attribute predictions in chest CT images. The proposed approach has been evaluated on the 2632 nodules in the public Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset. The experimental results suggest that our higher-order transfer approach shows the superior predictive performance not only in the most of the semantic attributes compared with the schemes of learning from scratch and the first-order transfer but also for the pathologic attribute compared with the related studies. In addition, we demonstrate an attribute transfer graph to reveal which attributes combination can supply the most useful information to boost the predictive performance of target attributes.

In [1], Baiying Lei was indicated as the corresponding author. Tianfu Wang and Baiying Lei should have been indicated as the corresponding authors.

Dermoscopy imaging analysis is a routine procedure for diagnosis and treatment of skin lesions. Segmentation is the very first step to demarcate skin lesions for further quantitative analysis. However, it is a challenging task due to various changes from different viewpoints and scales of skin lesions. To handle these challenges, we devise a new dense deconvolutional network (DDN) for skin lesion segmentation based on encoding module and decoding module. Our devised network consists of convolution unit, dense deconvolutionallayer (DDL) and chained residual pooling block. DDL is adopted to restore the high resolution of the original input by upsampling, while the chained residual pooling is utilized to fuse multilevel features. Also, the hierarchical supervision is added to capture low level detailed boundary information. The DDN is trained in an end-to-end manner and free of prior knowledge and complicated post-processing procedures. With fusing the local and global contextual information, the high-resolution prediction output is obtained. The validation on the public ISBI 2016 and 2017 skin lesion challenge dataset demonstrates the effectiveness of our proposed method.

Head circumference (HC) is one of the most important biometrics in assessing fetal growth during prenatal ultrasound examinations. However, the manual measurement of this biometric by doctors often requires substantial experience. We developed a learning-based framework that used prior knowledge and employed a fast ellipse fitting method (ElliFit) to measure HC automatically. We first integrated the prior knowledge about the gestational age and ultrasound scanning depth into a random forest classifier to localize the fetal head. We further used phase symmetry to detect the center line of the fetal skull and employed ElliFit to fit the HC ellipse for measurement. The experimental results from 145 HC images showed that our method had an average measurement error of 1.7 mm and outperformed traditional methods. The experimental results demonstrated that our method shows great promise for applications in clinical practice.

Ultrasound (US) imaging is a widely used screening tool for obstetric examination and diagnosis. Accurate acquisition of fetal standard planes with key anatomical structures is very crucial for substantial biometric measurement and diagnosis. However, the standard plane acquisition is a labor-intensive task and requires operator equipped with a thorough knowledge of fetal anatomy. Therefore, automatic approaches are highly demanded in clinical practice to alleviate the workload and boost the examination efficiency. The automatic detection of standard planes from US videos remains a challenging problem due to the high intraclass and low interclass variations of standard planes, and the relatively low image quality. Unlike previous studies which were specifically designed for individual anatomical standard planes, respectively, we present a general framework for the automatic identification of different standard planes from US videos. Distinct from conventional way that devises hand-crafted visual features for detection, our framework explores in- and between-plane feature learning with a novel composite framework of the convolutional and recurrent neural networks. To further address the issue of limited training data, a multitask learning framework is implemented to exploit common knowledge across detection tasks of distinctive standard planes for the augmentation of feature learning. Extensive experiments have been conducted on hundreds of US fetus videos to corroborate the better efficacy of the proposed framework on the difficult standard plane detection problem.

The quality of ultrasound (US) images for the obstetric examination is crucial for accurate biometric measurement. However, manual quality control is a labor intensive process and often impractical in a clinical setting. To improve the efficiency of examination and alleviate the measurement error caused by improper US scanning operation and slice selection, a computerized fetal US image quality assessment (FUIQA) scheme is proposed to assist the implementation of US image quality control in the clinical obstetric examination. The proposed FUIQA is realized with two deep convolutional neural network models, which are denoted as L-CNN and C-CNN, respectively. The L-CNN aims to find the region of interest (ROI) of the fetal abdominal region in the US image. Based on the ROI found by the L-CNN, the C-CNN evaluates the image quality by assessing the goodness of depiction for the key structures of stomach bubble and umbilical vein. To further boost the performance of the L-CNN, we augment the input sources of the neural network with the local phase features along with the original US data. It will be shown that the heterogeneous input sources will help to improve the performance of the L-CNN. The performance of the proposed FUIQA is compared with the subjective image quality evaluation results from three medical doctors. With comprehensive experiments, it will be illustrated that the computerized assessment with our FUIQA scheme can be comparable to the subjective ratings from medical doctors.

The gap between the computational and semantic features is the one of major factors that bottlenecks the computer-aided diagnosis (CAD) performance from clinical usage. To bridge this gap, we exploit three multi-task learning (MTL) schemes to leverage heterogeneous computational features derived from deep learning models of stacked denoising autoencoder (SDAE) and convolutional neural network (CNN), as well as hand-crafted Haar-like and HoG features, for the description of 9 semantic features for lung nodules in CT images. We regard that there may exist relations among the semantic features of "spiculation", "texture", "margin", etc., that can be explored with the MTL. The Lung Image Database Consortium (LIDC) data is adopted in this study for the rich annotation resources. The LIDC nodules were quantitatively scored w.r.t. 9 semantic features from 12 radiologists of several institutes in U.S.A. By treating each semantic feature as an individual task, the MTL schemes select and map the heterogeneous computational features toward the radiologists' ratings with cross validation evaluation schemes on the randomly selected 2400 nodules from the LIDC dataset. The experimental results suggest that the predicted semantic scores from the three MTL schemes are closer to the radiologists' ratings than the scores from single-task LASSO and elastic net regression methods. The proposed semantic attribute scoring scheme may provide richer quantitative assessments of nodules for better support of diagnostic decision and management. Meanwhile, the capability of the automatic association of medical image contents with the clinical semantic terms by our method may also assist the development of medical search engine.

Accurate segmentation of cervical cells in Pap smear images is an important step in automatic pre-cancer identification in the uterine cervix. One of the major segmentation challenges is overlapping of cytoplasm, which has not been well-addressed in previous studies. To tackle the overlapping issue, this paper proposes a learning-based method with robust shape priors to segment individual cell in Pap smear images to support automatic monitoring of changes in cells, which is a vital prerequisite of early detection of cervical cancer. We define this splitting problem as a discrete labeling task for multiple cells with a suitable cost function. The labeling results are then fed into our dynamic multi-template deformation model for further boundary refinement. Multi-scale deep convolutional networks are adopted to learn the diverse cell appearance features. We also incorporated high-level shape information to guide segmentation where cell boundary might be weak or lost due to cell overlapping. An evaluation carried out using two different datasets demonstrates the superiority of our proposed method over the state-of-the-art methods in terms of segmentation accuracy.

The goal of connectomics is to manifest the interconnections of neural system with the Electron Microscopy (EM) images. However, the formidable size of EM image data renders human annotation impractical, as it may take decades to fulfill the whole job. An alternative way to reconstruct the connectome can be attained with the computerized scheme that can automatically segment the neuronal structures. The segmentation of EM images is very challenging as the depicted structures can be very diverse.To address this difficult problem, a deep contextual network is proposed here by leveraging multi-level contextual information from the deep hierarchical structure to achieve better segmentation performance.To further improve the robustness against the vanishing gradients and strengthen the capability of the back-propagation of gradient flow, auxiliary classifiers are incorporated in the architecture of our deep neural network. It will be shown that our method can effectively parse the semantic meaning from the images with the underlying neural network and accurately delineate the structural boundaries with the reference of low-level contextual cues. Experimental results on the benchmark dataset of 2012 ISBI segmentation challenge of neuronal structures suggest that the proposed method can outperform the state-of-the-art methods by a large margin with respect to different evaluation measurements. Our method can potentially facilitate the automatic connectome analysis from EM images with less human intervention effort.

Registration and fusion of magnetic resonance (MR) and 3D transrectal ultrasound (TRUS) images of the prostate gland can provide high-quality guidance for prostate interventions. However, accurate MR-TRUS registration remains a challenging task, due to the great intensity variation between two modalities, the lack of intrinsic fiducials within the prostate, the large gland deformation caused by the TRUS probe insertion, and distinctive biomechanical properties in patients and prostate zones. To address these challenges, a personalized model-to-surface registration approach is proposed in this study. The main contributions of this paper can be threefold. First, a new personalized statistical deformable model (PSDM) is proposed with the finite element analysis and the patient-specific tissue parameters measured from the ultrasound elastography. Second, a hybrid point matching method is developed by introducing the modality independent neighborhood descriptor (MIND) to weight the Euclidean distance between points to establish reliable surface point correspondence. Third, the hybrid point matching is further guided by the PSDM for more physically plausible deformation estimation. Eighteen sets of patient data are included to test the efficacy of the proposed method. The experimental results demonstrate that our approach provides more accurate and robust MR-TRUS registration than state-of-the-art methods do. The averaged target registration error is 1.44 mm, which meets the clinical requirement of 1.9 mm for the accurate tumor volume detection. It can be concluded that the presented method can effectively fuse the heterogeneous image information in the elastography, MR, and TRUS to attain satisfactory image alignment performance.

This paper proposes a general boundary delineation method for 2D serial US images by modeling the spatial-temporal dynamics and generating the object boundary in each slice under the cell-based MAP scheme. The modeling of the spatial-temporal dynamics can serve as a prior to guide the cell-based MAP process and potentially maintain the contextual coherence. Experiments have been conducted on 8 sets of compression breast series and 5 sets of freehand breast acquisitions. The computer-generated results by our algorithm are compared to manual delineations prepared by experts. The experimental results suggest that the boundaries of the proposed method are not significantly different to manual outlines and are quite stable in terms of reproducibility.

Diffusion-weighted imaging is a specific MRI modality which is important for diagnosing brain infarction. In this study, we aim to find out that by using different b-values in DWI on 3T MRI for better diagnosis of acute or sub acute brain infarction. In 40 Patients with acute or sub acute brain infarction, we use the b-value 1000s/mm2and 1500s/mm2to process diffusion weighted images, and then these images were transferred to the imaging workstation for processing. We analyze the signal intensity and apparent diffusion coefficient (ADC) in the 1) infracted brain, 2) contra lateral normal brain, 3) background area of the infracted brain by circling the region of interest(ROI). It is shown that the higher b-value (1500s/mm2compared to 1000s/mm2) yield lower signal intensity and ADC value in both the normal brain tissue and the infracted area, The signal to noise ratio (SNR) and contrast to noise ratio (CNR) also decrease as the b-value increases, Applying diffusion weighted imaging generated by the 3T MRI machine different b-values(1000s/mm2and 1500s/mm2) to diagnose the brain infraction, we obtain better CNR and SNR by applying b-values 1000s/mm2with statistically significant differences.

As a potential biomarker for women's cardiovascular and chronic kidney diseases, breast arterial calcification (BAC) in mammography has become an emerging research topic in recent years. To provide more objective measurement for vascular structures with calcium depositions in mammography, a new computerized method is introduced in this paper to delineate the calcified vessels. Specifically, we leverage two underlying cues, namely calcification and vesselness, into a multiple seeded tracking with uncertainty scheme. This new vessel-tracking scheme generates plenty of sampling paths to describe the complicated topology of the vascular structures with calcium depositions. A compiling and linking process is further carried out to organize the sampling paths together to be the vessel segments that likely belong to the same vessel tract. The proposed method has been evaluated on 63 mammograms, by comparison with manual delineations from two experts using various assessment metrics. The experiment results confirm the efficacy and stability of the proposed method, and also indicate that the proposed method can be potentially used as a convenient BAC measurement tool in replacement of the trivial and tedious manual delineation tasks.