Accurate prediction of pathological subtypes on radiological images is one of the most important deep learning (DL) tasks for the appropriate selection of clinical treatment. It is challenging for conventional DL models to obtain sufficient pathological labels for training because of the heavy workload, invasive surgery, and knowledge requirements in pathological analysis. However, existing methods based on limited annotations, such as active learning (AL) and semi-supervised learning (SSL), have difficulty in capturing lesion's effective features because of the complicated semantic information of radiologic images. In this article, we introduce an efficient domain knowledge-guided semantic prediction framework that integrates domain knowledge-guided AL and SSL methods. This framework can effectively predict pathological subtypes on the basis of radiologic images with limited pathological annotations via three key modules: 1) the discriminative spatial-semantic feature extraction module captures the spatial-semantic features of lesions as semantic information that can better reflect the semantic relationship and effectively mitigate overfitting risk; 2) the explicit sign-guided anchor attention module measures the multimodal semantic distribution of samples under the guidance of clinical domain knowledge, thus selecting the most representative AL samples for pathological labeling; and 3) the implicit radiomics-guided dual-task entanglement module exploits the inherent constraint relationships between implicit radiomics features (IRFs) and pathological subtypes, facilitating the aggregation of unlabeled data. Experiments have been extensively conducted to evaluate our method in two clinical tasks: the pathological grading prediction in pancreatic neuroendocrine neoplasms (pNENs) and muscular invasiveness prediction in bladder cancer (BCa). The experimental results on both tasks demonstrate that the proposed method consistently outperforms the state-of-the-art approaches by a large margin.

Differential diagnosis of tumors is important for computer-aided diagnosis. In computer-aided diagnosis systems, expert knowledge of lesion segmentation masks is limited as it is only used during preprocessing or as supervision to guide feature extraction. To improve the utilization of lesion segmentation masks, this study proposes a simple and effective multitask learning network that improves medical image classification using self-predicted segmentation as guiding knowledge; we call this network RS$^{2}$-net. In RS$^{2}$-net, the predicted segmentation probability map obtained from the initial segmentation inference is added to the original image to form a new input, which is then reinput to the network for the final classification inference. We validated the proposed RS$^{2}$-net using three datasets: the pNENs-Grade dataset, which tested the prediction of pancreatic neuroendocrine neoplasm grading, and the HCC-MVI dataset, which tested the prediction of microvascular invasion of hepatocellular carcinoma, and ISIC 2017 public skin lesion dataset. The experimental results indicate that the proposed strategy of reusing self-predicted segmentation is effective, and RS$^{2}$-net outperforms other popular networks and existing state-of-the-art studies. Interpretive analytics based on feature visualization demonstrates that the improved classification performance of our reuse strategy is due to the semantic information that can be acquired in advance in a shallow network.

Approximately 13%–40% of patients with Crohn's disease (CD) show a primary loss of response to infliximab (IFX) therapy. Therefore, differentiating potential responders from primary nonresponders is clinically important. In this double-center study, we developed and validated a computed tomography enterography (CTE)-based radiomic signature (RS) for identification of CD patients at high risk of primary nonresponse (PNR) to IFX therapy, and demonstrated its incremental value to the clinical model. A total of 244 patients (training cohort, n = 119; test cohort 1, n = 51; test cohort 2, n = 74) were retrospectively recruited. Their clinical data and pretreatment CTE were retrieved and analyzed. All patients underwent IFX induction therapy. Reliability of clinical factors and radiomic-based features were assessed with the area under the receiver operating characteristic curve (AUC). In all, 1130 radiomic features were extracted from the whole inflamed gut in CTE images. In training cohort and test cohorts 1 and 2, the RS that discriminated PNR to IFX therapy yielded AUCs of 0.848, 0.789, and 0.789, respectively (all p < 0.05). By combining the clinical predictors (C-reactive protein, albumin, and body mass index) and RS, the radiomic-clinical model showed an increase in predicting performance (AUCs: 0.864, 0.794, and 0.791, respectively; all p < 0.05). Decision curve analysis and net reclassification improvement demonstrated the clinical usefulness of the radiomic-clinical model. In this study, the proposed RS showed potential as a clinical aid for the accurate identification of CD patients at high risk of PNR to IFX therapy before treatment. A combination of the RS and existing clinical factors might enable a step forward precise medicine.

Due to the lack of properly annotated medical data, exploring the generalization capability of the deep model is becoming a public concern. Zero-shot learning (ZSL) has emerged in recent years to equip the deep model with the ability to recognize unseen classes. However, existing studies mainly focus on natural images, which utilize linguistic models to extract auxiliary information for ZSL. It is impractical to apply the natural image ZSL solutions directly to medical images, since the medical terminology is very domain-specific, and it is not easy to acquire linguistic models for the medical terminology. In this work, we propose a new paradigm of ZSL specifically for medical images utilizing cross-modality information. We make three main contributions with the proposed paradigm. First, we extract the prior knowledge about the segmentation targets, called relation prototypes, from the prior model and then propose a cross-modality adaptation module to inherit the prototypes to the zero-shot model. Second, we propose a relation prototype awareness module to make the zero-shot model aware of information contained in the prototypes. Last but not least, we develop an inheritance attention module to recalibrate the relation prototypes to enhance the inheritance process. The proposed framework is evaluated on two public cross-modality datasets including a cardiac dataset and an abdominal dataset. Extensive experiments show that the proposed framework significantly outperforms the state of the arts.

Unsupervised domain adaption has proven to be an effective approach for alleviating the intensive workload of manual annotation by aligning the synthetic source-domain data and the real-world target-domain samples. Unfortunately, mapping the target-domain distribution to the source-domain unconditionally may distort the essential structural information of the target-domain data. To this end, we firstly propose to introduce a novel multi-anchor based active learning strategy to assist domain adaptation regarding the semantic segmentation task. By innovatively adopting multiple anchors instead of a single centroid, the source domain can be better characterized as a multimodal distribution, thus more representative and complimentary samples are selected from the target domain. With little workload to manually annotate these active samples, the distortion of the target-domain distribution can be effectively alleviated, resulting in a large performance gain. The multi-anchor strategy is additionally employed to model the target-distribution. By regularizing the latent representation of the target samples compact around multiple anchors through a novel soft alignment loss, more precise segmentation can be achieved. Extensive experiments are conducted on public datasets to demonstrate that the proposed approach outperforms state-of-the-art methods significantly, along with thorough ablation study to verify the effectiveness of each component. The code will be released soon at https://github.com/munanning/MADA.

Current clinical practice or radiomics studies of pancreatic neuroendocrine neoplasms (pNENs) require manual delineation of the lesions in computed tomography (CT) images, which is time-consuming and subjective. We used a semi-automatic deep learning (DL) method for segmentation of pNENs and verified its feasibility in radiomics analysis. This retrospective study included two datasets: Dataset 1, contrast-enhanced CT images (CECT) of 80 and 18 patients respectively collected from two centers; and Dataset 2, CECT of 56 and 16 patients respectively from two centers. A DL-based semi-automatic segmentation model was developed and validated with Dataset 1 and Dataset 2, and the segmentation results were used for radiomics analysis from which the performance was compared against that based on manual segmentation. The mean Dice similarity coefficient of the trained segmentation model was 81.8% and 74.8% for external validation with Dataset 1 and Dataset 2 respectively. Four classifiers frequently used in radiomics studies were trained and tested with leave-one-out cross-validation strategy. For pathological grading prediction with Dataset 1, the area under the receiver operating characteristic curve (AUC) with semi-automatic segmentation was up to 0.76 and 0.87 respectively for internal and external validation. For recurrence study with Dataset 2, the AUC with semi-automatic segmentation was up to 0.78. All these AUCs were not statistically significant from the corresponding results based on manual segmentation. Our study showed that DL-based semi-automatic segmentation is accurate and feasible for the radiomics analysis in pNENs.

Recently, an emerging trend in medical image classification is to combine radiomics framework with deep learning classification network in an integrated system. Although this combination is efficient in some tasks, the deep learning-based classification network is often difficult to capture an effective representation of lesion regions, and prone to face the challenge of overfitting, leading to unreliable features and inaccurate results, especially when the sizes of the lesions are small or the training dataset is small. In addition, these combinations mostly lack an effective feature selection mechanism, which makes it difficult to obtain the optimal feature selection. In this paper, we introduce a novel and effective deep semantic segmentation feature-based radiomics (DSFR) framework to overcome the above-mentioned challenges, which consists of two modules: the deep semantic feature extraction module and the feature selection module. Specifically, the extraction module is utilized to extract hierarchical semantic features of the lesions from a trained segmentation network. The feature selection module aims to select the most representative features by using a novel feature similarity adaptation algorithm. Experiments are extensively conducted to evaluate our method in two clinical tasks: the pathological grading prediction in pancreatic neuroendocrine neoplasms (pNENs), and the prediction of thrombolytic therapy efficacy in deep venous thrombosis (DVT). Experimental results on both tasks demonstrate that the proposed method consistently outperforms the state-of-the-art approaches by a large margin.