Accurate prediction of pathological subtypes on radiological images is one of the most important deep learning (DL) tasks for the appropriate selection of clinical treatment. It is challenging for conventional DL models to obtain sufficient pathological labels for training because of the heavy workload, invasive surgery, and knowledge requirements in pathological analysis. However, existing methods based on limited annotations, such as active learning (AL) and semi-supervised learning (SSL), have difficulty in capturing lesion's effective features because of the complicated semantic information of radiologic images. In this article, we introduce an efficient domain knowledge-guided semantic prediction framework that integrates domain knowledge-guided AL and SSL methods. This framework can effectively predict pathological subtypes on the basis of radiologic images with limited pathological annotations via three key modules: 1) the discriminative spatial-semantic feature extraction module captures the spatial-semantic features of lesions as semantic information that can better reflect the semantic relationship and effectively mitigate overfitting risk; 2) the explicit sign-guided anchor attention module measures the multimodal semantic distribution of samples under the guidance of clinical domain knowledge, thus selecting the most representative AL samples for pathological labeling; and 3) the implicit radiomics-guided dual-task entanglement module exploits the inherent constraint relationships between implicit radiomics features (IRFs) and pathological subtypes, facilitating the aggregation of unlabeled data. Experiments have been extensively conducted to evaluate our method in two clinical tasks: the pathological grading prediction in pancreatic neuroendocrine neoplasms (pNENs) and muscular invasiveness prediction in bladder cancer (BCa). The experimental results on both tasks demonstrate that the proposed method consistently outperforms the state-of-the-art approaches by a large margin.

Differential diagnosis of tumors is important for computer-aided diagnosis. In computer-aided diagnosis systems, expert knowledge of lesion segmentation masks is limited as it is only used during preprocessing or as supervision to guide feature extraction. To improve the utilization of lesion segmentation masks, this study proposes a simple and effective multitask learning network that improves medical image classification using self-predicted segmentation as guiding knowledge; we call this network RS$^{2}$-net. In RS$^{2}$-net, the predicted segmentation probability map obtained from the initial segmentation inference is added to the original image to form a new input, which is then reinput to the network for the final classification inference. We validated the proposed RS$^{2}$-net using three datasets: the pNENs-Grade dataset, which tested the prediction of pancreatic neuroendocrine neoplasm grading, and the HCC-MVI dataset, which tested the prediction of microvascular invasion of hepatocellular carcinoma, and ISIC 2017 public skin lesion dataset. The experimental results indicate that the proposed strategy of reusing self-predicted segmentation is effective, and RS$^{2}$-net outperforms other popular networks and existing state-of-the-art studies. Interpretive analytics based on feature visualization demonstrates that the improved classification performance of our reuse strategy is due to the semantic information that can be acquired in advance in a shallow network.

Diagnosis of tumors is an important direction of computer-aided diagnosis (CAD). The shape, size, and boundary of the tumor are widely-used diagnostic evidence, and the corresponding segmentation annotated by the radiologists is a vital expert knowledge, which can be used as supervision to guide feature extraction. Therefore, this study firstly introduces a multi-task learning (MTL) network integrating segmentation task for predicting grading of pancreatic neuroendocrine neoplasms (pNENs) and the microvascular invasion (MVI) of hepatocellular carcinoma (HCC). The proposed network combines a powerful split-attention-based encoder and a U-net decoder, and achieves the best performance in comparisons of other popular networks and previous studies. In addition, feature map visualization suggests that the reason for the improved classification performance may be that MTL makes the encoder pay more attention to lesions and extract more semantic information.

Current clinical practice or radiomics studies of pancreatic neuroendocrine neoplasms (pNENs) require manual delineation of the lesions in computed tomography (CT) images, which is time-consuming and subjective. We used a semi-automatic deep learning (DL) method for segmentation of pNENs and verified its feasibility in radiomics analysis. This retrospective study included two datasets: Dataset 1, contrast-enhanced CT images (CECT) of 80 and 18 patients respectively collected from two centers; and Dataset 2, CECT of 56 and 16 patients respectively from two centers. A DL-based semi-automatic segmentation model was developed and validated with Dataset 1 and Dataset 2, and the segmentation results were used for radiomics analysis from which the performance was compared against that based on manual segmentation. The mean Dice similarity coefficient of the trained segmentation model was 81.8% and 74.8% for external validation with Dataset 1 and Dataset 2 respectively. Four classifiers frequently used in radiomics studies were trained and tested with leave-one-out cross-validation strategy. For pathological grading prediction with Dataset 1, the area under the receiver operating characteristic curve (AUC) with semi-automatic segmentation was up to 0.76 and 0.87 respectively for internal and external validation. For recurrence study with Dataset 2, the AUC with semi-automatic segmentation was up to 0.78. All these AUCs were not statistically significant from the corresponding results based on manual segmentation. Our study showed that DL-based semi-automatic segmentation is accurate and feasible for the radiomics analysis in pNENs.

Recently, an emerging trend in medical image classification is to combine radiomics framework with deep learning classification network in an integrated system. Although this combination is efficient in some tasks, the deep learning-based classification network is often difficult to capture an effective representation of lesion regions, and prone to face the challenge of overfitting, leading to unreliable features and inaccurate results, especially when the sizes of the lesions are small or the training dataset is small. In addition, these combinations mostly lack an effective feature selection mechanism, which makes it difficult to obtain the optimal feature selection. In this paper, we introduce a novel and effective deep semantic segmentation feature-based radiomics (DSFR) framework to overcome the above-mentioned challenges, which consists of two modules: the deep semantic feature extraction module and the feature selection module. Specifically, the extraction module is utilized to extract hierarchical semantic features of the lesions from a trained segmentation network. The feature selection module aims to select the most representative features by using a novel feature similarity adaptation algorithm. Experiments are extensively conducted to evaluate our method in two clinical tasks: the pathological grading prediction in pancreatic neuroendocrine neoplasms (pNENs), and the prediction of thrombolytic therapy efficacy in deep venous thrombosis (DVT). Experimental results on both tasks demonstrate that the proposed method consistently outperforms the state-of-the-art approaches by a large margin.