Accurate prediction of drug-target interactions (DTI) is pivotal in drug discovery. However, existing approaches often fail to capture deep intra-modal feature in-teractions or achieve effective cross-modal alignment, limiting predictive performance and generalization. To address these challenges, we propose M3ST-DTI, a multi-task learning model that enables multi-stage integration and alignment of multi-modal features for DTI prediction. M3ST-DTI incorporates three types of features-textual, structural, and functional and enhances intra-modal representations using self-attention mechanisms and a hybrid pooling graph attention module. For early-stage feature alignment and fusion, the model integrates MCA with Gram loss as a structural constraint. In the later stage, a BCA module captures fine-grained interactions between drugs and targets within each modality, while a deep orthogonal fusion module mitigates feature redundancy. Extensive evaluations on benchmark datasets demonstrate that M3ST-DTI consistently outperforms state-of-the-art methods across diverse metrics.

HER2 is a critical biomarker for the diagnosis and prognosis of breast cancer. However, accurately predicting HER2 status from pathological images remains challenging due to tumor heterogeneity and the complexity of molecular mechanisms. Existing methods that integrate clinical and pathological features often struggle with structural semantic discretization, resulting in cross-modal information loss and semantic gaps. We propose a cross-modal adaptive attention collaboration neural network (CMCNet) to address these limitations. The proposed model features a heterogeneous dual-branch architecture to align and fuse pathological images and clinical data. A hierarchical Transformer architecture is employed to extract both local cell-level and global tissue-level semantic features from pathology images, while a large language model is used to capture sequential semantic information from clinical features. To further enhance integration, a cross-modal adaptive attention mechanism is introduced to regulate feature fusion dynamically. We validated the model using 556 samples from the TCGA-BRCA dataset. Experimental results demonstrate that CMCNet achieves a maximum accuracy of 84.26% in predicting HER2 status, outperforming current state-of-the-art methods. Ablation studies confirm the contribution of each module, and visualization analyses offer new insights into HER2 status diagnosis and prognosis prediction.

The incidence of breast cancer is increasing rapidly around the world. Accurate classification of the breast cancer subtype from hematoxylin and eosin images is the key to improve the precision of treatment. However, the high consistency of disease subtypes and uneven distribution of cancer cells seriously affect the performance of multi-classification methods. Furthermore, it is difficult to apply existing classification methods to multiple datasets. In this article, we propose a collaborative transfer network (CTransNet) for multi-classification of breast cancer histopathological images. CTransNet consists of a transfer learning backbone branch, a residual collaborative branch, and a feature fusion module. The transfer learning branch adopts the pre-trained DenseNet structure to extract image features from ImageNet. The residual branch extracts target features from pathological images in a collaborative manner. The feature fusion strategy of optimizing these two branches is used to train and fine-tune CTransNet. Experiments show that CTransNet achieves 98.29% classification accuracy on the public BreaKHis breast cancer dataset, exceeding the performance of state-of-the-art methods. Visual analysis is carried out under the guidance of oncologists. Based on the training parameters of the BreaKHis dataset, CTransNet achieves superior performance on other two public breast cancer datasets (breast-cancer-grade-ICT and ICIAR2018_BACH_Challenge), indicating that CTransNet has good generalization performance.

Clinical studies have proved that both structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) are implicitly associated with neuropsychiatric disorders (NDs), and integrating multi-modal to the binary classification of NDs has been thoroughly explored. However, accurately classifying multiple classes of NDs remains a challenge due to the complexity of disease subclass. In our study, we develop a heterogeneous neural network (H-Net) that integrates sMRI and fMRI modes for classifying multi-class NDs. To account for the differences between the two modes, H-Net adopts a heterogeneous neural network strategy to extract information from each mode. Specifically, H-Net includes an multi-layer perceptron based (MLP-based) encoder, a graph attention network based (GAT-based) encoder, and a cross-modality transformer block. The MLP-based and GAT-based encoders extract semantic features from sMRI and features from fMRI, respectively, while the cross-modality transformer block models the attention of two types of features. In H-Net, the proposed MLP-mixer block and cross-modality alignment are powerful tools for improving the multi-classification performance of NDs. H-Net is validate on the public dataset (CNP), where H-Net achieves 90% classification accuracy in diagnosing multi-class NDs. Furthermore, we demonstrate the complementarity of the two MRI modalities in improving the identification of multi-class NDs. Both visual and statistical analyses show the differences between ND subclasses.

Segmenting stroke lesions and assessing the thrombolysis in cerebral infarction (TICI) grade are two important but challenging prerequisites for an auxiliary diagnosis of the stroke. However, most previous studies have focused only on a single one of two tasks, without considering the relation between them. In our study, we propose a simulated quantum mechanics-based joint learning network (SQMLP-net) that simultaneously segments a stroke lesion and assesses the TICI grade. The correlation and heterogeneity between the two tasks are tackled with a single-input double-output hybrid network. SQMLP-net has a segmentation branch and a classification branch. These two branches share an encoder, which extracts and shares the spatial and global semantic information for the segmentation and classification tasks. Both tasks are optimized by a novel joint loss function that learns the intra- and inter-task weights between these two tasks. Finally, we evaluate SQMLP-net with a public stroke dataset (ATLAS R2.0). SQMLP-net obtains state-of-the-art metrics (Dice:70.98% and accuracy:86.78%) and outperforms single-task and existing advanced methods. An analysis found a negative correlation between the severity of TICI grading and the accuracy of stroke lesion segmentation.

Stroke is a cerebrovascular disease with high mortality and disability rates. The occurrence of the stroke typically produces lesions of different sizes, with the accurate segmentation and detection of small-size stroke lesions being closely related to the prognosis of patients. However, the large lesions are usually correctly identified, the small-size lesions are usually ignored. This article provides a hybrid contextual semantic network (HCSNet) that can accurately and simultaneously segment and detect small-size stroke lesions from magnetic resonance images. HCSNet inherits the advantages of the encoder-decoder architecture and applies a novel hybrid contextual semantic module that generates high-quality contextual semantic features from the spatial and channel contextual semantic features through the skip connection layer. Moreover, a mixing-loss function is proposed to optimize HCSNet for unbalanced small-size lesions. HCSNet is trained and evaluated on 2D magnetic resonance images produced from the Anatomical Tracings of Lesions After Stroke challenge (ATLAS R2.0). Extensive experiments demonstrate that HCSNet outperforms several other state-of-the-art methods in its ability to segment and detect small-size stroke lesions. Visualization and ablation experiments reveal that the hybrid semantic module improves the segmentation and detection performance of HCSNet.

Identifying the subtypes of low-grade glioma (LGG) can help prevent brain tumor progression and patient death. However, the complicated non-linear relationship and high dimensionality of 3D brain MRI limit the performance of machine learning methods. Therefore, it is important to develop a classification method that can overcome these limitations. This study proposes a self-attention similarity-guided graph convolutional network (SASG-GCN) that uses the constructed graphs to complete multi-classification (tumor-free (TF), WG, and TMG). In the pipeline of SASG-GCN, we use a convolutional deep belief network and a self-attention similarity-based method to construct the vertices and edges of the constructed graphs at 3D MRI level, respectively. The multi-classification experiment is performed in a two-layer GCN model. SASG-GCN is trained and evaluated on 402 3D MRI images which are produced from the TCGA-LGG dataset. Empirical tests demonstrate that SASG-GCN accurately classifies the subtypes of LGG. The accuracy of SASG-GCN achieves 93.62%, outperforming several other state-of-the-art classification methods. In-depth discussion and analysis reveal that the self-attention similarity-guided strategy improves the performance of SASG-GCN. The visualization revealed differences between different gliomas.

Neural networks (NNs) have demonstrated exciting results on various tasks within the last decade. For example, the performance on image classification tasks has been improved dramatically. However, the performance evaluations are often based on a black-box performance, such as accuracy, while insightful analysis of the black-box, such as the prediction formation mechanism, is often missing. Empirically, a NN usually produces a stable overall performance on the same task across multiple training trials when treating it as a black-box. However, when unveiling the black-box, the performance is usually volatile. The decision-making criteria learned by the training trials are often significantly different, which is problematic in many ways. We believe achieving consistent criteria between different training trials is equally important to achieving high performance, if not more. This work, firstly, evaluates the decision-making criteria of NNs via inputs sensitivity using feature-attribution explanation methods in combination with computational analysis and clustering analysis. Through intensive experimentation, we find that decision-making criteria are easily distinguishable between training trials of the same architecture and task, suggesting the criteria learned between training trials are significantly inconsistent. To mitigate this inconsistency, we propose three general training schemes. Our demonstration result shows that the proposed methods effectively reduce the inconsistency of the decision-making criteria learned by different training trials while maintaining the overall performance.

Accurate and rapid diagnosis of coronavirus disease 2019 (COVID-19) from chest CT scans is of great importance and urgency during the worldwide outbreak. However, radiologists have to distinguish COVID-19 pneumonia from other pneumonia in a large number of CT scans, which is tedious and inefficient. Thus, it is urgently and clinically needed to develop an efficient and accurate diagnostic tool to help radiologists to fulfill the difficult task. In this study, we proposed a deep supervised autoencoder (DSAE) framework to automatically identify COVID-19 using multi-view features extracted from CT images. To fully explore features characterizing CT images from different frequency domains, DSAE was proposed to learn the latent representation by multi-task learning. The proposal was designed to both encode valuable information from different frequency features and construct a compact class structure for separability. To achieve this, we designed a multi-task loss function, which consists of a supervised loss and a reconstruction loss. Our proposed method was evaluated on a newly collected dataset of 787 subjects including COVID-19 pneumonia patients, other pneumonia patients, and normal subjects without abnormal CT findings. Extensive experimental results demonstrated that our proposed method achieved encouraging diagnostic performance and may have potential clinical application for the diagnosis of COVID-19.

Magnetic resonance imagings (MRIs) are providing increased access to neuropsychiatric disorders that can be made available for advanced data analysis. However, the single type of data limits the ability of psychiatrists to distinguish the subclasses of this disease. In this paper, we propose an ensemble hybrid features selection method for the neuropsychiatric disorder classification. The method consists of a 3D DenseNet and a XGBoost, which are used to select the image features from structural MRI images and the phenotypic feature from phenotypic records, respectively. The hybrid feature is composed of image features and phenotypic features. The proposed method is validated in the Consortium for Neuropsychiatric Phenomics (CNP) dataset, where samples are classified into one of the four classes (healthy controls (HC), attention deficit hyperactivity disorder (ADHD), bipolar disorder (BD), and schizophrenia (SD)). Experimental results show that the hybrid feature can improve the performance of classification methods. The best accuracy of binary and multi-class classification can reach 91.22 and 78.62 percent, respectively. We analyze the importance of phenotypic features and image features in different classification tasks. The importance of the structure MRI images is highlighted by incorporating phenotypic features with image features to generate hybrid features. We also visualize the features of three neuropsychiatric disorders and analyze their locations in the brain region.

Medical image segmentation is the primary measure of medical image analysis. With the development of deep learning, U-net based approaches have presented for different medical image segmentation tasks. However, the pooling and the simple convolution operation for deep feature maps in the U-shaped network would lead to the coarse segmentation result. In this paper, we design a local adaptive U-net (LA U-net) for medical image segmentation. There are two major modules: the Local Adaptive Module (LAM) and Multi-scale Convolution Module (MCM) in the network. The LAM get more feature maps from each down-sampling process. The MCM capture more global information for the encoding path. To validate the proposed network's performance, we verify it on two datasets: DRIVE dataset, and ISIC 2018 dataset; the results show that LA U-net achieves superior performance on two datasets.

The context-based convolutional neural network (CNN) is one of the most well-known CNNs to improve the performance of semantic segmentation. It has achieved remarkable success in various medical image segmentation tasks. However, extracting rich and useful context information from complex and changeable medical images is a challenge for medical image segmentation. In this study, a novel Multi-Receptive-Field CNN (MRFNet) is proposed to tackle this challenge. MRFNet offers the optimal receptive field for each subnet in the encoder-decoder module (EDM) and generates multi-receptive-field context information at the feature map level. Moreover, MRFNet fuses these multi-feature maps by the concatenation operation. MRFNet is evaluated on 3 public medical image data sets, including SISS, 3DIRCADb, and SPES. Experimental results show that MRFNet achieves the outstanding performance on all 3 data sets, and outperforms other segmentation methods on 3DIRCADb test set without pre-training the model.

Accurate segmentation of glioma from 3D medical images is vital to numerous clinical endpoints. While manual segmentation is subjective and time-consuming, fully automated extraction is quite imperative and challenging due to the intrinsic heterogeneity of tumor structures. In this study, we propose a multi-level glioma segmentation framework, 3D Residual-Attention-Atrous U-Net (RAAU-Net), using 3D U-Net combined attention mechanism with atrous convolution. The 3D RAAU-Net can extract contextual information by combining low- and high-resolution feature maps. The attention mechanism is embedded in each skip connection layer of 3D RAAU-Net to enhance feature representations. Meanwhile, the atrous convolution is adopted in the whole network architecture to incorporate large and rich semantic information. Furthermore, we design a new training scheme to reduce false positives and enhance generalization. Eventually, our proposed segmentation method is evaluated on the validation dataset from the Multimodal Brain Tumor Image Segmentation Challenge (BraTS) 2018 and achieve a competitive result with average Dice score of 88% for the whole tumor, 79% for the tumor core and 73% for the enhancing tumor, respectively. Quantitative results and visual analysis have proven that these improvements in 3D RAAU-Net are effective and achieve a better segmentation accuracy compared with the baseline.

A tentative diagnosis is a preliminary suspicion of patient status, which is usually made by physicians according to patient narrative right at admission. It largely depends on the experiences and professional knowledge of physicians. We explored a combination model for automatic tentative diagnosis prediction based on clinical narratives. Text features are extracted in two ways. Firstly, the context semantic features are extracted by attention-based bidirectional long-short term memory (BiLSTM) network. Secondly, the symptom concepts recognized from input texts by Metamap and are vectorized by TF-IDF. Two combination strategies are proposed to utilize both two features for one candidate international classification of diseases (ICD) code recommendation: feature vectors combination and prediction results combination. The experiments performed on MIMIC III dataset. Both of the two combination strategies achieved better performance, comparing with either of the model based on single type feature.

Osteoarthritis (OA) is the most common type of arthritis. Analgesics are widely used in the process of the treatment of arthritis. Analgesics are particularly used by OA patients which may increase the risk of cardiovascular disease by 20% to 50% overall. In this study, we proposed an interpretable model to predict side effects of analgesics on cardiovascular disease for OA patients. One task of our study is to predict whether OA patients can use analgesics. We weighed accuracy and interpretability among state-of-the-art methods, and constructed a non-linear model by the Gradient Boosting Decision Tree technique. The AUC of the prediction model was 0.96. Another task was to select informative risk features (RFs) by our proposed model. We sought to identify risk features in literature from the biomedical. Most of the selected RFs are validated by the medical literature and some new RFs could attract the interest across the medical research. The performance of the proposed model, showed its superiority compared with well-known machine learning algorithms in terms of AUC.