Brain disorders are associated with impairments in cognitive and social functioning, placing a substantial burden on families, healthcare systems, and communities. However, accurate diagnosis remains challenging due to complex higher order interactions among brain regions. Existing graph-based methods are largely limited to pairwise connectivity. In addition, these methods often fail to fully exploit well-established neuroscientific prior knowledge, resulting in limited biological interpretability and suboptimal diagnostic performance. Therefore, we propose a prior knowledge-guided adaptive hypergraph learning (PK-AHGL) framework that represents individual-level functional connectivity networks as hypergraphs to capture higher order multiregion interactions while incorporating neuroscientific prior knowledge for brain disorder diagnosis. PK-AHGL consists of three key modules: 1) an adaptive hypergraph convolution module. Unlike traditional hypergraph neural networks that use static hyperedge weights, this module adaptively learns the weights of different hyperedges; 2) a sparse affinity Laplacian module. Key brain regions are extracted from disorder related functional brain networks and used as prior knowledge. Based on these regions, we compute a hyperedge similarity matrix that encourages similar hyperedges to have similar weights; and 3) a proportional margin ranking module. This module further utilizes prior knowledge by guiding hyperedges containing a higher proportion of key brain regions to obtain larger weights. Experiments on autism brain imaging data exchange (ABIDE), Strategic Research Program for the Promotion of Brain Science (SRPBS)-schizophrenia (SCZ), SRPBS-major depressive disorder (MDD), and Alzheimer’s disease neuroimaging initiative (ADNI) show that PK-AHGL achieves accuracies of 75.78%, 82.66%, 74.89%, and 77.22%, respectively, outperforming multiple state-of-the-art methods. These results suggest that PK-AHGL provides an effective auxiliary tool for brain disorder diagnosis and may support community-oriented mental health services.

Survival risk prediction is crucial for personalized treatment of gliomas. Medical image foundational models can explore complex medical features, which are critical for prognosis in gliomas. We propose SAM-Risk, which uses a customized SAM-Med3D with multi-view representation fusion and clinical knowledge-based age-grade stratified loss for glioma survival risk prediction. First, to utilize potential interactions between multiple views at an early stage, we design a 3D representation generation module that transforms 1D handcrafted radiomics and clinical features into 3D representations, which are fused with multimodal MRIs through a multi-view representation fusion module. The fused representation is fed into the customized SAM-Med3D, fine-tuned using LoRA and a disparity function to extract survival risk-related features. We design a feature refinement module to explore the inter-channel relationships among the outputs of the fine-tuned SAM-Med3D. Additionally, we propose an age-grade stratified loss based on glioma prognosis standards to make the predicted risk more consistent with clinical prior knowledge. Validated on two publicly available UCSF-PDGM and BraTS2020 datasets, SAM-Risk achieves a C-index of 75.08% and 73.67%, respectively, outperforming several survival risk prediction methods.

Accurate glioma grading using noninvasive multiparametric MRI is essential to minimize biopsy risks and guide treatment planning. Current AI-based approaches struggle to prioritize 3D pathological features due to the lack of shape-sensitive multi-dimensional spatial features and exhibit channel redundancy, while multi-scale fusion lacks semantic patch (MRI subregion) relationships. To address these limitations, we propose a Wavelet Multi-Dimensional and Mamba-Guided Semantic Graph Feature Fusion Network (WM-MSGFF-Net) for glioma grading that integrates two innovative modules. First, the WaveletAdaptive Feature Refinement (WAFR) module processes features through the Adaptive Wavelet Multi-Dimensional Collaboration (AWMDC) sub-module, utilizing wavelet-based convolutions to extract joint shape-sensitive spatial features across depth, height, and width dimensions, coupled with adaptive channel attention to capture sparse pathological features. Subsequently, WAFR applies Self-attention to model patch-level dependencies. Second, the Semantic-Aware Multi-Scale Feature Fusion (SAMSFF) module constructs glioma-specific graphs from WAFR and encoderderived patches, using cosine similarity to define edges and leveraging Mamba-enhanced Graph Convolutional Networks (GCNs) to establish semantic relationships. Experiments on publicly available datasets demonstrate that WM-MSGFF-Net achieves an accuracy of 95.7 % on BraTS2020 and 95.6 % on UCSF-PDGM, outperforming 8 baseline methods. By integrating effective volumetric feature extraction with semantic patch relationships, WM-MSGFF-Net offers a promising approach to noninvasive glioma grading that could support diagnostic decision-making.

Accurately predicting isocitrate dehydrogenase (IDH) mutations is crucial for glioma diagnosis, but the limited availability of multimodal MRI restricts the generalization of existing methods. Fine-tuning foundation models is a common solution, yet their lack of domain-specific knowledge often impairs optimal performance. Clinical studies show that both the imagemodal T2-FLAIR mismatch sign knowledge and text-modal demographic information are useful for IDH genotyping. Thus, we propose a novel network that integrates multimodal clinical knowledge to guide the fine-tuning of the multimodal foundation model M3D for glioma IDH genotyping on multimodal MRI. To fully utilize multimodal knowledge, we first extract multigranularity T2-FLAIR mismatch features from different layers via a Mixture-of-Experts (MoE) pool-based mismatch adapter (incorporating an MoE-based pooling mechanism and a spatialchannel mismatch attention module). Meanwhile, demographic information is converted into text prompts and encoded by the M3D text encoder to generate demographic-based text embeddings. In the encoder, T2-FLAIR mismatch features are integrated at the end of each ViT Block to introduce imagingspecific knowledge, while text features are further processed via a cross-modal text-image attention fusion adapter to enhance representation learning in the joint feature space. We evaluated the approach on an internal dataset (from 3 public datasets, 871 patients) and an independent external dataset (501 patients). It achieved 93.25 % accuracy on the internal dataset and 86.03 % on the external dataset, with only$\mathbf{2. 6 7 M}$trainable parameters, outperforming 7 existing state-of-the-art IDH genotyping methods.

Deformable image registration is fundamental in medical image analysis. Existing pyramid-based deep learning methods suffer from coarse deformation decomposition, poor inter-level transitions, and error accumulation. To address these, we propose GMReg, an unsupervised pyramid network for medical image registration based on Mamba correlation. GMReg introduces intra-level multi-scale decomposition: each pyramid level splits deformation fields into subfields with different receptive fields via grouping, using Group Mamba correlation layers for feature matching/fusion, and convolutional prediction for sub-fields. A channel attention-based context fusion module enhances inter-group interaction, while a multi-scale edge enhancement module guides subfield fusion to improve boundary sensitivity. Experiments on two public brain MRI datasets (LPBA40, Mind-Boggle) show GMReg significantly outperforms state-of-the-art methods in registration accuracy. Additionally, results on the FIRE dataset demonstrate that GMReg also holds potential for the fundus image registration task.

The Alberta Stroke Program Early CT Score (AS-PECTS) is a systematic method for assessing the extent of early ischemic changes on non-contrast CT (NCCT) of patients with acute ischemic stroke (AIS). The ASPECTS regions are anatomically and physiologically interconnected, making them suitable for analysis by graph neural networks. However, most existing methods fail to effectively use the relationships and bilateral differences. This study designs an Anatomical Relationship and Isomorphic Graph approximation guided Graph Convolutional Network for ASPECTS scoring on NCCT. Firstly, we propose a node construction guided by anatomical structures, i.e., utilizing the region anatomical relationship adjacency matrix of the ASPECTS regions to build the nodes. Secondly, to optimize the information propagation among nodes, we propose isomorphic graph approximation, utilizing edge learning, connectivity-based subgraph selection, and supervised isomorphic subgraph approximation to supervise the construction of isomorphic subgraphs. We validate our method on private AIS datasets which included NCCT scans of 257 AIS patients. The results show that the proposed method achieves interclass correlation coefficients of 0.8554 for total ASPECTS, and accuracy of 90.61% for dichotomized ASPECTS scoring (4), outperforming 9 state-of-the-art methods.

Segmentation of lesions in Acute Ischemic Stroke (AIS) patients on Non-Contrast CT (NCCT) scans is pivotal for expedited diagnosis and effective treatment planning. The subtle and 4.5-hour golden treatment window characteristic of AIS lesions on NCCT makes fully automated segmentation more preferred. We introduce AISCT-SAM, a fine-tuned Segment Anything Model (SAM), for this task. Specifically, we: 1) integrate an adaptive gate control mechanism with Adapter and Low-Rank Adaptation to comprehensively fine-tune the encoder of the foundational SAM model; 2) incorporate a depth adapter to extract richer 3D contextual information, enhancing lesion continuity; 3) propose a self-prompt generator that leverages bilateral hemisphere differences to generate prompts for the mask decoder, enabling fully automatic segmentation. It is evaluated on both public and private datasets, with results from internal and external validations showing state-of-the-art performance. Additionally, volumetric analysis indicates that AISCT-SAM can provide valuable reference for diagnostic and treatment decisions. Our data and code are publicly available at: https://github.com/GitHub-TXZ/AISCT-SAM.

The use of estimators instead of stochastic gradients for updates has been shown to improve algorithm convergence rates of, but their impact on generalization remains under-explored. In this paper, we investigate how estimators influence generalization. Our focus is on two widely studied problems: stochastic optimization (SO) and stochastic compositional optimization (SCO), both under convex and non-convex settings. For SO problems, we first analyze the generalization error of the STORM algorithm as a foundational step. We then extend our analysis to SCO problems by introducing an algorithmic framework that encompasses several popular algorithmic approaches. Through this framework, we conduct a generalization analysis, uncovering new insights into the impact of estimators on generalization. Subsequently, we provide a detailed analysis of three specific algorithms within this framework: SCGD, SCSC, and COVER, to explore the effects of different estimator strategies. Furthermore, in the context of SCO, we propose a novel definition of stability and a new decomposition of excess risk in the non-convex setting. Our analysis indicates two key findings: (1) In SCO problems, eliminating the estimator for the gradient of the inner function does not impact generalization performance while significantly reducing computational and storage overhead. (2) Faster convergence rates are consistently associated with better generalization performance.

Whole Slide Image (WSI) representation is critical for cancer subtyping, cancer recognition and mutation prediction.Training an end-to-end WSI representation model poses significant challenges, as a standard gigapixel slide can contain tens of thousands of image tiles, making it difficult to compute gradients of all tiles in a single mini-batch due to current GPU limitations. To address this challenge, we propose a method of dynamic residual encoding with slide-level contrastive learning (DRE-SLCL) for end-to-end WSI representation. Our approach utilizes a memory bank to store the features of tiles across all WSIs in the dataset. During training, a mini-batch usually contains multiple WSIs. For each WSI in the batch, a subset of tiles is randomly sampled and their features are computed using a tile encoder. Then, additional tile features from the same WSI are selected from the memory bank. The representation of each individual WSI is generated using a residual encoding technique that incorporates both the sampled features and those retrieved from the memory bank. Finally, the slide-level contrastive loss is computed based on the representations and histopathology reports ofthe WSIs within the mini-batch. Experiments conducted over cancer subtyping, cancer recognition, and mutation prediction tasks proved the effectiveness of the proposed DRE-SLCL method.

Multimodal Biomedical Image Incremental Learning (MBIIL) is essential for handling diverse tasks and modalities in the biomedical domain, as training separate models for each modality or task significantly increases inference costs. Existing incremental learning methods focus on task expansion within a single modality, whereas MBIIL seeks to train a unified model incrementally across modalities. The MBIIL faces two challenges: I) How to preserve previously learned knowledge during incremental updates? II) How to effectively leverage knowledge acquired from existing modalities to support new modalities? To address these challenges, we propose MSLoRA-CR, a method that fine-tunes Modality-Specific LoRA modules while incorporating Contrastive Regularization to enhance intra-modality knowledge sharing and promote inter-modality knowledge differentiation. Our approach builds upon a large vision-language model (LVLM), keeping the pretrained model frozen while incrementally adapting new LoRA modules for each modality or task. Experiments on the incremental learning of biomedical images demonstrate that MSLoRA-CR outperforms both the state-of-the-art (SOTA) approach of training separate models for each modality and the general incremental learning method (incrementally fine-tuning LoRA). Specifically, MSLoRA-CR achieves a 1.88% improvement in overall performance compared to unconstrained incremental learning methods while maintaining computational efficiency. Our code is publicly available at https://github.com/VentusAislant/MSLoRA_CR.

Graph learning-based multi-modal integration and classification is one of the most challenging tasks for disease prediction. To effectively offset the negative impact among modalities in the process of multi-modal integration and heterogeneous information extractions from graphs, we propose a novel method called Multi-modal Multi-Kernel Graph Learning (MMKGL). To solve the problem of negative impact among modalities, we propose a multi-modal graph embedding module to construct a multi-modal graph. Different from conventional methods that manually construct static graphs for all modalities, each modality generates a separate graph by adaptive learning, where a function graph and a supervision graph are introduced for optimization during the multi-graph fusion embedding process. We then propose a multi-kernel graph learning module to extract heterogeneous information from the multi-modal graph. The information in the multi-modal graph at different levels is aggregated by convolutional kernels with different receptive field sizes, followed by generating a cross-kernel discovery tensor for disease prediction. Our method is evaluated on the benchmark Autism Brain Imaging Data Exchange (ABIDE) dataset and outperforms the state-of-the-art methods. In addition, discriminative brain regions associated with autism are identified by our model, providing guidance for the study of autism pathology.

Accurate patient outcome predictions are essential for healthcare improvement, yet utilizing diverse medical data raises privacy and security concerns. Federated learning enables collaborative model training while preserving data privacy. However, the heterogeneity of clinical features among hospitals poses a challenge, leading to suboptimal performance in conventional federated learning. In response, we propose FedComDist, a personalized federated learning approach designed to maximize the use of heterogeneous features across hospitals for patient outcome prediction. Our approach incorporates a novel method for optimizing common global and distinct local parameters. We categorize input clinical features into two main groups-common and distinct-based on their presence across all hospitals and decouple the model parameters into common and distinct accordingly. The common features are used to train common global parameters, which are aggregated and optimized on the server, making them trainable across all hospitals. Meanwhile, the distinct features are used to train local parameters and optimized using the local dataset of each hospital. Our approach is evaluated on the eICU dataset, a publicly available multi-center clinical dataset, to predict patient clinical outcomes, specifically mortality and Length of Stay (LoS). The experimental results demonstrate the effectiveness of our approach compared to various federated learning methods and provide enhanced privacy through parameter decoupling.

The electrocardiogram (ECG) serves as a crucial tool for myocardial infarction (MI) localization, and deep learning methods have proven effective in assisting physicians with MI localization. Traditional MI localization methods are purely data-driven, and the quality of the data significantly affects the model's performance, particularly in the localization of rare MI. We propose a knowledge-driven graph representation learning (KD-GRL) framework which is designed to guide deep learning models in identifying key features for MI localization using prior knowledge. The MI localization knowledge graph (KG) is constructed by integrating medical knowledge about MI localization, including ECG leads and morphological manifestations, the correlations between MI localization labels, diagnostic rules, and patient demographic information. KG effectively represents the relationships among various entities, which include ECG signal entities, morphological feature entities, and demographic feature entities. The embeddings of these entities are obtained using parallel patient multi-feature extractors. Additionally, a KG aggregation method based on edge relation projection (ERP) is proposed to aggregate the relational information in the MI localization KG. Ultimately, the MI localization task is transformed into a link prediction task between patient entity and localization label entities within the KG. We conduct experiments on two public datasets, PTB and PTBXL, achieving F1-scores of 48.90% and 46.06%, respectively, both surpassing the comparison methods. Additionally, due to the incorporation of diagnostic knowledge, our method outperforms the comparison methods in localizing rare MIs.

Pathological diagnosis assists in saving human lives, but such models are annotation hungry and pathological images are notably expensive to annotate. Contrastive learning could be a promising solution that relies only on the unlabeled training data to generate informative representations. However, the majority of current methods in contrastive learning have the following two issues: (1) positive samples produced through random augmentation are less challenging, and (2) false negative pairs problem caused by negative sampling bias. To alleviate the above issues, we propose a novel contrastive learning method called Cluster-Aware Adversarial Contrastive Learning (CA2CL). Specifically, a mixed data augmentation technique is provided to learn more transferable representations by generating more discriminative sample pairs. Furthermore, to mitigate the effects of inherent false negative pairs, we adopt a cluster-aware loss to identify similarities between instances and incorporate them into the process of contrastive learning. Finally, we generate challenging contrastive data pairs by adversarial learning, and adversarially learn robust representations in the representation space without the labeled training data, which aims to maximize the similarity between the augmented sample and the related adversarial sample. Our proposed CA2CL is evaluated on two public datasets: NCT-CRC-HE and PCam for the fine-tuning and linear evaluation tasks and on two other public datasets: GlaS and CARG for the detection and segmentation tasks, respectively. Extensive experimental results demonstrate the superior performance improvement of our method over several Self-supervised learning (SSL) methods and ImageNet pretraining particularly in scenarios with limited data availability for all four tasks.

Pathological question answering (PQA) is vital in computational pathology, as it involves interpreting pathological images and answering questions posed by humans. This interaction offers an effective means of engaging with users and enhancing the understanding of pathology-related information. Recent methods developed using large vision-language models (LVLMs), such as QUILT-LLAVA, have made significant progress in advancing PQA. However, existing models, such as those using the QUILT-1M dataset, neglect the quality of the training set during the fine-tuning stage, leading to sub-optimal performance. We recognize that high-quality training data can significantly enhance model performance. Therefore, we design a model-based data filtering strategy to remove images with obvious impurities from the instruction fine-tuning dataset. The filtered high-quality images are then used to fine-tune the model. Additionally, most existing vision-language alignment strategies focus primarily on aligning local features through next-word prediction, leading to a relatively homogeneous granularity in inter-modal alignment. To address this issue, we propose a global-wise alignment module, which introduces global-level contrastive learning during the pretraining stage to establish multi-granularity alignment between pathological images and language descriptions. Based on the above two processes, we design our method, named Global Contrastive Learning with High-Quality Data (GCL-HQD) for pathological question answering in LVLMs. Extensive experiments on two types of experimental settings demonstrate the effectiveness of the GCL-HQD method.

Resting-state Functional Magnetic Resonance Imaging (rs-fMRI) has been widely used in Mild Cognitive Impairment (MCI) detection. However, most existing methods rely on fixed sliding windows to capture the temporal variations in brain connectivity and fail to model temporal features effectively. To address these limitations, we propose a spatio-temporal graph Transformer framework. First, we segment rs-fMRI data into distinct segments using the multivariate Gaussian distribution of brain region signals and remove redundant segments based on cumulative Jensen-Shannon divergence. Next, we extract features for each brain region and time point within segments, using these temporal features to guide spatial feature aggregation. A position encoder is designed to leverage temporal features across segments to capture both local and global spatio-temporal information. Finally, we employ a population graph framework as the classifier, with spatio-temporal features and demographic data of each subject forming the edge weights, to output the detection results. Our method achieves 92.80%, 81.89%, and 82.80% accuracy in MCI detection, early MCI detection, and late MCI detection tasks on the ADNI3 and ADNI2 datasets, respectively, outperforming existing methods.

The diagnosis of mild cognitive impairment (MCI), which is an early stage of Alzheimer’s disease (AD), has great clinical significance. Medical imaging and gene sequencing technologies have provided sufficient multimodality data for MCI diagnostic studies. However, how to effectively extract the rich representations from multimodality data remains a challenging task. To address this challenging task, we propose a new multimodality multiview graph representations and knowledge embedding (MMGK) framework to diagnose MCI. First, to obtain rich information from multimodality data, we extract multiview feature representations from magnetic resonance imaging (MRI) and genetic data. Afterward, considering the correlations between subjects, all subjects are constructed into a graph based on the different single-view feature representations, respectively. To further enrich the correlations between subjects, demographic data are utilized through knowledge embedding. Finally, to perform MCI diagnosis on multiview graphs, graph convolutional networks (GCNs) are utilized. In addition, to further improve the performance of MCI diagnosis, a two-step ensemble learning method is proposed. The proposed framework is evaluated on 188 subjects from the AD Neuroimaging Initiative (ADNI). Experimental results show that our proposed framework achieves good performance with accuracy reaching 0.888, and outperforms some state-of-the-art (SOTA) methods. In addition, the proposed framework is applied to Parkinson’s disease (PD) diagnosis and achieves 0.856 accuracy. Overall, our proposed method has potential for clinical application in MCI diagnosis and other diseases via integrating MRI, genetic data, and demographic data. Our code is available at:https://github.com/miacsu/MMGK.

Lesion segmentation is a fundamental step for the diagnosis of acute ischemic stroke (AIS). Non-contrast CT (NCCT) is still a mainstream imaging modality for AIS lesion measurement. However, AIS lesion segmentation on NCCT is challenging due to low contrast, noise and artifacts. To achieve accurate AIS lesion segmentation on NCCT, this study proposes a hybrid convolutional neural network (CNN) and Transformer network with circular feature interaction and bilateral difference learning. It consists of parallel CNN and Transformer encoders, a circular feature interaction module, and a shared CNN decoder with a bilateral difference learning module. A new Transformer block is particularly designed to solve the weak inductive bias problem of the traditional Transformer. To effectively combine features from CNN and Transformer encoders, we first design a multi-level feature aggregation module to combine multi-scale features in each encoder and then propose a novel feature interaction module containing circular CNN-to-Transformer and Transformer-to-CNN interaction blocks. Besides, a bilateral difference learning module is proposed at the bottom level of the decoder to learn the different information between the ischemic and contralateral sides of the brain. The proposed method is evaluated on three AIS datasets: the public AISD, a private dataset and an external dataset. Experimental results show that the proposed method achieves Dices of 61.39% and 46.74% on the AISD and the private dataset, respectively, outperforming 17 state-of-the-art segmentation methods. Besides, volumetric analysis on segmented lesions and external validation results imply that the proposed method is potential to provide support information for AIS diagnosis.

Medical image denoising, as a part of medical image processing, is significant for the assessment and diagnosis of diseases. To improve the medical image denoising performance of existing deep learning methods, we propose a lightweight plug-and-play module (LP2M) with low complexity, which can be plugged before current image denoising methods. Specifically, the proposed LP2M contains three stacked Convolutional Neural Network (CNN) based blocks: a image receptor block, an adaptive receptive field selection block and a high-low frequency processing block. The image receptor block can perceive color or grayscale images and perform preliminary processing. The adaptive receptive field selection block includes two parallel paths with different receptive fields (i.e., convolution kernel sizes) and the adaptive weighting operation, which can process multi-scale information in the image. The high-low frequency processing block consists of a low frequency pathway using convolutional layers with large kernel sizes, and a high frequency pathway using covolutional layers with small kernel sizes, which can process the low and high frequency components in images. Extensive validation experiments are performed on five state-of-the-art denoising methods on multiple medical image datasets for three different medical image denoising tasks (X-ray image denoising, magnetic resonance image denoising and dermoscopic image denoising). Experimental results show that our proposed LP2M can effectively improve the results of these five state-of-the-art methods for three denoising tasks with only increasing 0.996K parameters and 63.112M FLOPs, and it can provide potential direction for improving image denoising.

Self-supervised learning provides a possible solution to extract effective visual representations from unlabeled pathological images. However, most of the existing methods either do not effectively utilize domain-specific information or are designed and optimized for image classification, resulting in these pre-trained models that may not be optimal for pathological image segmentation. In this paper, we propose DKSL: Domain-specific Knowledge guided Self-supervised Learning, which uses image reconstruction tasks to aid contrastive learning and exploits single-dye stained pathological images after stain separation as domain-specific knowledge to guide the model. Our method provides a novel way to exploit the domain-specific knowledge of pathological images. In contrastive learning, we add single-dye stained images as an expansion of the original positive samples to the contrastive learning process to preserve more global semantic information. In image reconstruction, the model is forced to focus on local image details relevant to downstream tasks by reconstructing single-dye stained images from the representation extracted by the encoder of contrastive learning. Finally, the encoder and decoder from the pre-training stage are fine-tuned by the downstream segmentation task. Fine-tuning experimental results demonstrate that DKSL outperforms state-of-the-art methods with Dices of 90.50% and 79.68% on two publicly available datasets, GlaS and MoNuSeg, respectively.

Isocitrate dehydrogenase (IDH) is a key molecular feature for gliomas, and the prediction of IDH is also an important task for computer-aided diagnosis using magnetic resonance imaging (MRI). To address this changllenge, we introduce a multi-modal MRI-based characteristics inspired network for IDH Genotyping (M3CI-Net), which pay more attention to the different characteristics information of different MRI modalities T1, T2, T1ce, Flair. In M3CI-Net, a pre-fusion module with multi-channel attention mechanism is used to fuse T1ce and Flair modalities and capture as much as possible luminance and contrast information, and the edge information is obtained from T2 modality by using edge detection module. Finally, the feature information between modalities are fused and input into a CNN-Transformer based encoder structure to extract shared spatial and global information from multi-modal MRI, and the information of multiple scales frome encoder are input into the linear layer for IDH genotype classification after pooling, meanwhile, the CNN based decoder with skip-connection for glioma segmentation works for assisting IDH genotyping. Then, we proposed images’ pre-fusion loss, segmentation loss, IDH genotyping loss, and use uncertainty weight training method to balance the weights of these loss. we evaluate our proposed method on Brats2020, and achieve an acceracy of 0.88, an AUC of 0.94, a specificity of 0.92, a sensitivity of 0.84 in IDH genotyping, which is superior to the state-of-the-art methods.

Accurately predicting survival of esophageal cancer is essential for clinical precision treatment. However, the existing region of interest (ROI) based methods not only require prior medical knowledge to complete the delineation of tumor, but may also lead to excessive sensitivity of the model towards ROI. To address these challenges, we design a fully automated CT-guided learning that combines a CNN-Transformer size aware U-Net and a ranked survival prediction network together to automatically predict the survival of patients with esophageal cancer. Specifically, we first incorporate the Transformer with shifted windowing multi-head self-attention mechanism into the base of the encoder in the U-Net to capture the long-range dependency in the 3D CT images. Then, to alleviate the imbalance between the ROI and the background in CT images, we design a size-aware coefficient for the segmentation loss. Finally, we design a ranked pair sorting loss to learn more fully the ranked information hidden in esophageal cancer patients. To validate the effectiveness of our method, we conduct extensive experiments on a dataset containing 759 esophageal cancer samples. The experimental results demonstrate that our proposed method can still achieve the best performance in survival prediction without ROI ground truth.

Medical image segmentation is indispensable for diagnosis and prognosis of many diseases. To improve the segmentation performance, this study proposes a new 2D body and edge aware network with multi-scale short-term concatenation for medical image segmentation. Multi-scale short-term concatenation modules which concatenate successive convolution layers with different receptive fields, are proposed for capturing multi-scale representations with fewer parameters. Body generation modules with feature adjustment based on weight map computing via enlarging the receptive fields, and edge generation modules with multi-scale convolutions using Sobel kernels for edge detection, are proposed to separately learn body and edge features from convolutional features in decoders, making the proposed network be body and edge aware. Based on the body and edge modules, we design parallel body and edge decoders whose outputs are fused to achieve the final segmentation. Besides, deep supervision from the body and edge decoders is applied to ensure the effectiveness of the generated body and edge features and further improve the final segmentation. The proposed method is trained and evaluated on six public medical image segmentation datasets to show its effectiveness and generality. Experimental results show that the proposed method achieves better average Dice similarity coefficient and 95% Hausdorff distance than several benchmarks on all used datasets. Ablation studies validate the effectiveness of the proposed multi-scale representation learning modules, body and edge generation modules and deep supervision.

Cervical abnormal cell detection is a challenging task as the morphological discrepancies between abnormal and normal cells are usually subtle. To determine whether a cervical cell is normal or abnormal, cytopathologists always take surrounding cells as references to identify its abnormality. To mimic these behaviors, we propose to explore contextual relationships to boost the performance of cervical abnormal cell detection. Specifically, both contextual relationships between cells and cell-to-global images are exploited to enhance features of each region of interest (RoI) proposal. Accordingly, two modules, dubbed as RoI-relationship attention module (RRAM) and global RoI attention module (GRAM), are developed and their combination strategies are also investigated. We establish a strong baseline by using Double-Head Faster R-CNN with a feature pyramid network (FPN) and integrate our RRAM and GRAM into it to validate the effectiveness of the proposed modules. Experiments conducted on a large cervical cell detection dataset reveal that the introduction of RRAM and GRAM both achieves better average precision (AP) than the baseline methods. Moreover, when cascading RRAM and GRAM, our method outperforms the state-of-the-art (SOTA) methods. Furthermore, we show that the proposed feature-enhancing scheme can facilitate image- and smear-level classification.

Recent methods of patient clinical outcome prediction focus on embedding the temporal time-series data by sequential data encoders without considering the dependency between the different variables and the static demographics data. To solve this problem and achieve better patient outcome prediction, we propose an attention-based memory fusion (AMF) network with Gated Recurrent Unit (GRU) (called GRU-AMFN) to model the dependency between the different time-series and static demographic data and extract effective personalized representation about the patient’s clinical health status. We evaluate our proposed GRU-AMFN method on eICU, a publicly available dataset, to validate its effectiveness for the in-hospital mortality prediction task. Experimental results demonstrate that our proposed method outperforms several state-of-the-art models for the in-hospital mortality prediction task. Ablation studies show the effectiveness of the proposed attention-based memory fusion module and the adaptive fusion module. Besides, our proposed method finds several static demographic and time-series features that are important for mortality prediction.

Ki67 is a promising molecular biomarker for the diagnosis of lung adenocarcinoma. However, previous methods to determine Ki67 status often require tumor tissue sampling, which is invasive for patients. This study proposes a multi-view signature fusion network (MVSF), combining deep learning encoded (DLE) signatures, handcrafted radiomics (HCR) signatures, and clinical information to noninvasively predict Ki67 status. Multi-view signatures are combined through a tensor fusion network to obtain potentially high-dimensional signatures. Finally, a cooperative game theory-based approach is applied to quantitatively interpret the contribution of signatures to decision-making. The proposed MVSF is evaluated on a retrospectively collected dataset of 661 patients. Experimental results show that the MVSF achieves encouraging performance, with an area under the receiver operating characteristic curve of 0.80 and an accuracy of 0.78, outperforming several state-of-the-art Ki67 status prediction methods, which implies that our proposed method could provide potential support for Ki67 status prediction.

Resting-state functional magnetic resonance imaging (rs-fMRI) images have been widely used for diagnosis of schizophrenia. With rs-fMRI, most existing schizophrenia diagnostic methods have revealed schizophrenia’s functional abnormalities from the following three scales, i.e., regional neural activity alterations, functional connectivity abnormalities and brain network dysfunctions. However, many schizophrenia diagnosis methods do not consider the fusion of features from the three scales. In this study, we propose a schizophrenia diagnostic method based on multi-scale feature representation and ensemble learning. Firstly, features including the three scales (region, connectivity and network) are extracted from rs-fMRI images using the brainnetome atlas. For each scale, feature selection, i.e., least absolute shrinkage and selection operator, is applied to identify effective sub-features related to schizophrenia classification by a grid search. Then the selected sub-features of each scale are input to support vector machine with linear kernel to classify schizophrenia patients and healthy controls respectively. To further improve the schizophrenia diagnostic performance, an ensemble learning framework named E-RCN is proposed to average the probabilities obtained by the classifiers of each scale in decision level. By leave-one-out cross-validation on the center for biomedical research excellence dataset (COBRE), our proposed method achieves encouraging diagnosis performance, outperforming several state-of-the-art methods. In addition, ranked by the occurence frequency of each brain region within the leave-one-out cross-validation experiments, some brain regions related to schizophrenia, i.e., thalamus and middle temporal gyrus, and important elaborate subregions, i.e., Tha_L_8_8, MTG_L_4_4 and MTG_R_4_4, are found.

Effective fusion of multimodal magnetic resonance imaging (MRI) is of great significance to boost the accuracy of glioma grading thanks to the complementary information provided by different imaging modalities. However, how to extract the common and distinctive information from MRI to achieve complementarity is still an open problem in information fusion research. In this study, we propose a deep neural network model termed as multimodal disentangled variational autoencoder (MMD-VAE) for glioma grading based on radiomics features extracted from preoperative multimodal MRI images. Specifically, the radiomics features are quantized and extracted from the region of interest for each modality. Then, the latent representations of variational autoencoder for these features are disentangled into common and distinctive representations to obtain the shared and complementary data among modalities. Afterwards, cross-modality reconstruction loss and common-distinctive loss are designed to ensure the effectiveness of the disentangled representations. Finally, the disentangled common and distinctive representations are fused to predict the glioma grades, and SHapley Additive exPlanations (SHAP) is adopted to quantitatively interpret and analyze the contribution of the important features to grading. Experimental results on two benchmark datasets demonstrate that the proposed MMD-VAE model achieves encouraging predictive performance (AUC:0.9939) on a public dataset, and good generalization performance (AUC:0.9611) on a cross-institutional private dataset. These quantitative results and interpretations may help radiologists understand gliomas better and make better treatment decisions for improving clinical outcomes.

The accurate prediction of isocitrate dehydrogenase (IDH) mutation and glioma segmentation are important tasks for computer-aided diagnosis using preoperative multimodal magnetic resonance imaging (MRI). The two tasks are ongoing challenges due to the significant inter-tumor and intra-tumor heterogeneity. The existing methods to address them are mostly based on single-task approaches without considering the correlation between the two tasks. In addition, the acquisition of IDH genetic labels is expensive and costly, resulting in a limited number of IDH mutation data for modeling. To comprehensively address these problems, we propose a fully automated multimodal MRI-based multi-task learning framework for simultaneous glioma segmentation and IDH genotyping. Specifically, the task correlation and heterogeneity are tackled with a hybrid CNN-Transformer encoder that consists of a convolutional neural network and a transformer to extract the shared spatial and global information learned from a decoder for glioma segmentation and a multi-scale classifier for IDH genotyping. Then, a multi-task learning loss is designed to balance the two tasks by combining the segmentation and classification loss functions with uncertain weights. Finally, an uncertainty-aware pseudo-label selection is proposed to generate IDH pseudo-labels from larger unlabeled data for improving the accuracy of IDH genotyping by using semi-supervised learning. We evaluate our method on a multi-institutional public dataset. Experimental results show that our proposed multi-task network achieves promising performance and outperforms the single-task learning counterparts and other existing state-of-the-art methods. With the introduction of unlabeled data, the semi-supervised multi-task learning framework further improves the performance of glioma segmentation and IDH genotyping. The source codes of our framework are publicly available at https://github.com/miacsu/MTTU-Net.git.

Medical image segmentation is a fundamental step for diagnosis and prognosis. This study proposes a new body and edge aware network for automated 2D medical image segmentation (called BEA-SegNet). The proposed BEA-SegNet consists of a shared encoder, a body and edge decouple (BEdecouple) module, two parallel decoders for body and edge segmentation. In the encoder and decoders, short-term multi-scale concatenation (STMSC) modules are utilized to implement multi-scale representation. We design a BEdecouple module to decouple the convolutional features into the body and edge features, making the proposed method be body and edge aware. The body and edge decoders utilize Bedecouple modules in each level to learn more effective features for the body and edge segmentation respectively, and their outputs are fused to generate the final segmentation. Besides, the body and edge supervision are applied to improve the final segmentation. The proposed BEA-SegNet is trained and evaluated on the International Skin Imaging Collaboration challenge 2018 dataset (ISIC2018). Experimental results show that the proposed BEA-SegNet achieves an average Dice similarity coefficient of 90.3% and an average Hausdorff distance of 15.9 for the skin lesion segmentation task and outperforms five benchmarks for skin lesion segmentation.

Automatic detection of Alzheimer’s disease (AD) is conducive to intervention in the disease progression. MMSE score prediction can reveal the development of AD. In recent years, some studies have designed multi-task methods for AD detection and MMSE score prediction to take advantage of the correlation between them. However, how to use the correlation between the two task features is still a problem. To address this challenge, we propose a multi-task feature interactive leanrning network (MTFIL-Net) to perform AD detection and MMSE score prediction. First, we interact the features acquired by CNNs corresponding to the two tasks to take advantage of the feature correlation between the two tasks. The interaction module extracts the shared features of the two tasks and concatenate them with the features of the two task. Then, we design a joint loss based on cross entropy and smooth L1 function. We use the distribution of MMSE scores to dynamically adjust the relationship between the two tasks. We validate our method with subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). We use the ADNI1 dataset for training and testing, and used the ADNI2 dataset as an external validation set. Our proposed MTFIL-Net reached an ACC of 0.86 for AD detection and a correlation coefficient of 0.67 for MMSE score prediction on the ADNI1 dataset, and reached an ACC of 0.85 for AD detection and a correlation coefficient of 0.66 for MMSE score prediction on the ADNI2 dataset. Experiment results show that MTFIL-Net effectively utilizes the correlation between AD and MMSE score.

Automatic identification of adventitious respiratory sound has still been a challenging problem in recent years. To address this challenge, we propose an adventitious respiratory sound classification network (ARSC-Net), which combines residual block with channel-spatial attention for accurate classification. Specifically, we extract two types of features from adventitious respiratory sound, including Mel-Frequency Cepstral Coefficients (MFCCs) and Mel-spectrogram. The two types of features are entered into the parallel encoders paths with residual attention for extracting feature representation, and then fused into a channel-spatial attention module to adaptively focus on the important features between channel and spatial part for the classification task. Moreover, the channel-spatial attention can enhance the feature representation, in which the channel attention explores the inter-channel relationship of the spectrums, and then the inter-spatial correlation mapping is generated by the spatial attention introduced serially. We evaluate our proposed method on ICBHI 2017 database. Experimental results show that our proposed method achieves encouraging predictive performance with an accuracy of 80.0% for identifying abnormal sounds from normal sounds, and with an accuracy of 92.4% for distinguishing crackles from wheezes. In addition, our method also achieves a score of 56.76% for the four-class sound classification of adventitious sounds and outperforms several state-of-the-art methods.

In this paper, we relate the operation of image dynamic range adjustment to the following two tasks: 1) for a high dynamic range (HDR) image, its dynamic range will be mapped to the available dynamic range of display devices and 2) for a low dynamic range (LDR) image, its distribution of intensity will be extended to adequately utilize the full dynamic range of display devices. The common goal of both tasks is to preserve or even enhance the details and improve the visibility of scenes when being matched to the available dynamic range of a display device. In this paper, we propose an efficient method for image dynamic range adjustment with three adaptive steps. First, according to the histogram of the luminance map separated from the given RGB image, two suitable Gamma functions are adaptively selected to separately adjust the luminance of the dark and bright components. Second, an adaptive fusion strategy is proposed to combine the two adjusted luminance maps in order to balance the enhancement of the details in different regions. Third, an adaptive luminance-dependent color restoration method is designed to combine the fused luminance map with the original color components to obtain more consistent color saturation between the images before and after dynamic range adjustment. Extensive experiments show that the proposed method can efficiently compress the dynamic range of HDR scenes with good contrast, clear details, and high structural fidelity of the original image appearance. In addition, the proposed method can also obtain promising performance when being used to enhance LDR nighttime images and greatly facilitates the object (car) detection in nighttime traffic scenes.

Night-time vehicle detection is essential in building intelligent transportation systems (ITS) for road safety. Most of current night-time vehicle detection approaches focus on one or two classes of vehicles. In this paper, we present a novel multiclass vehicle detection system based on tensor decomposition and object proposal. Commonly used features such as histogram of oriented gradients and local binary pattern often produce useless image blocks (regions), which can result in unsatisfactory detection performance. Thus, we select blocks via feature ranking after tensor decomposition and only extract features from these selected blocks. To generate windows that contain all vehicles, we propose a novel object-proposal approach based on a state-of-the-art object-proposal method, local features, and image region similarity. The three terms are summed with learned weights to compute the reliability score of each proposal. A bio-inspired image enhancement method is used to enhance the brightness and contrast of input images. We have built a Hong Kong night-time multiclass vehicle dataset for evaluation. Our proposed vehicle detection approach can successfully detect four types of vehicles: 1) car; 2) taxi; 3) bus; and 4) minibus. Occluded vehicles and vehicles in the rain can also be detected. Our proposed method obtains 95.82% detection rate at 0.05 false positives per image, and it outperforms several state-of-the-art night-time vehicle detection approaches.

Object proposal is one of the most key pre-processing steps for nighttime vehicle detection systems in intelligent transportation systems. However, most current object proposal methods are developed on daytime data sets, and these methods demonstrate unsatisfactory results when they are used on nighttime images. Therefore, this paper presents a novel Bayes saliency-based object proposal generator for nighttime RGB traffic images to generate a modest and accurate set of proposals, which are more likely to be vehicles for preceding vehicle detection. First, we propose a new Bayes saliency detection approach in which prior estimation, feature extraction, weight estimation, and Bayes rule are used to compute saliency maps. Then, we propose a simple but effective object proposal generator based on the Bayes saliency map. Multi-scale sliding window, proposal rejecting, scoring, and non-maximum suppression are combined to generate a modest and effective set of proposals. Experimental results demonstrate that our proposed approach generates a modest set of proposals and outperforms some state-of-the-art methods on nighttime images in terms of various evaluation metrics. Furthermore, our proposed object proposal approach can improve the detection performance and the speed of several state-of-the-art vehicle detection approaches.

Detecting vehicle turn signals at night is critical for both assistant driving systems and autonomous driving systems. In this paper, we propose a novel method that consists of detection and tracking modules to achieve a high level of robustness. For nighttime vehicle detection, a Nakagami-image-based method is used to locate the regions containing vehicle lights. At the same time, a set of vehicle object proposals is generated using a region proposal network based on convolutional neural network (CNN) feature maps. Then, the light regions and proposals are combined to generate the regions of interest (ROIs) for the further detection. Vehicle candidates are extracted from the ROIs using a softmax classifier with CNN-based features. For the tracking module, we propose a perceptional hashing algorithm to track these vehicle candidates. During the tracking, turn signals are detected by analyzing the continuous intensity variation of the vehicle box sequences. Experimental results for typical sequences show that the proposed method can robustly detect and track a vehicle in front with over 95% accuracy and recognize the turning signals in night scenes with a detection rate of over 90%. The vehicle detection method improves the miss rate of state-of-the-art systems by more than 20%. In addition, the proposed vehicle tracking method outperforms other state-of-the-art systems.

This paper presents an effective nighttime vehicle detection system that combines a novel bioinspired image enhancement approach with a weighted feature fusion technique. Inspired by the retinal mechanism in natural visual processing, we develop a nighttime image enhancement method by modeling the adaptive feedback from horizontal cells and the center-surround antagonistic receptive fields of bipolar cells. Furthermore, we extract features based on the convolutional neural network, histogram of oriented gradient, and local binary pattern to train the classifiers with support vector machine. These features are fused by combining the score vectors of each feature with the learnt weights. During detection, we generate accurate regions of interest by combining vehicle taillight detection with object proposals. Experimental results demonstrate that the proposed bioinspired image enhancement method contributes well to vehicle detection. Our vehicle detection method demonstrates a 95.95% detection rate at 0.0575 false positives per image and outperforms some state-of-the-art techniques. Our proposed method can deal with various scenes including vehicles of different types and sizes and those with occlusions and in blurred zones. It can also detect vehicles at various locations and multiple vehicles.